[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:57 dist-test-slave-djkr krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:57 dist-test-slave-djkr krb5kdc[399](info): set up 2 sockets
Sep 28 23:16:57 dist-test-slave-djkr krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 23:16:58.579500 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:58 dist-test-slave-djkr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:16:58.684377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:58.760807 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:58.761000 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:58.761029 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:58.762176 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:58.762218 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:58.762238 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:58.762253 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:58.762269 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:58.763764 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:58.763994 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:58.764199 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:58.769093 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:58.769088 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:58.769229 412 server_base.cc:1034] running on GCE node
W20260928 23:16:58.769183 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:58.769528 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:58.769778 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:58.770905 412 hybrid_clock.cc:648] HybridClock initialized: now 1790637418770900 us; error 26 us; skew 500 ppm
Sep 28 23:16:58 dist-test-slave-djkr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:58.773141 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:16:58.773564 412 webserver.cc:460] Webserver started at http://127.0.98.62:41799/ using document root <none> and password file <none>
I20260928 23:16:58.773762 412 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:58.773803 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:58.773918 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:58.774889 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "53b82f2e5e5444d998a18197ca9fc605"
format_stamp: "Formatted at 2026-09-28 23:16:58 on dist-test-slave-djkr"
server_key: "de42660b1833e9c56ae87af9f9d044ab"
server_key_iv: "aafe227cf190a6b367cbb164f9911351"
server_key_version: "encryptionkey@0"
I20260928 23:16:58.775224 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "53b82f2e5e5444d998a18197ca9fc605"
format_stamp: "Formatted at 2026-09-28 23:16:58 on dist-test-slave-djkr"
server_key: "de42660b1833e9c56ae87af9f9d044ab"
server_key_iv: "aafe227cf190a6b367cbb164f9911351"
server_key_version: "encryptionkey@0"
I20260928 23:16:58.776523 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:16:58.777186 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:58.777315 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:16:58.777397 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "53b82f2e5e5444d998a18197ca9fc605"
format_stamp: "Formatted at 2026-09-28 23:16:58 on dist-test-slave-djkr"
server_key: "de42660b1833e9c56ae87af9f9d044ab"
server_key_iv: "aafe227cf190a6b367cbb164f9911351"
server_key_version: "encryptionkey@0"
I20260928 23:16:58.777453 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:58.785918 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:58.787456 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:58.787635 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:58.791009 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35111
I20260928 23:16:58.791047 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35111 every 8 connection(s)
I20260928 23:16:58.791318 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:58.792155 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:58.794222 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605: Bootstrap starting.
I20260928 23:16:58.794752 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:58.795008 480 log.cc:824] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:58.795640 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605: No bootstrap required, opened a new log
I20260928 23:16:58.797101 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b82f2e5e5444d998a18197ca9fc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35111 } }
I20260928 23:16:58.797228 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:58.797258 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b82f2e5e5444d998a18197ca9fc605, State: Initialized, Role: FOLLOWER
I20260928 23:16:58.797423 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b82f2e5e5444d998a18197ca9fc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35111 } }
I20260928 23:16:58.797485 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:58.797508 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:58.797552 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:58.798023 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b82f2e5e5444d998a18197ca9fc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35111 } }
I20260928 23:16:58.798111 480 leader_election.cc:302] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53b82f2e5e5444d998a18197ca9fc605; no voters:
I20260928 23:16:58.798254 480 leader_election.cc:288] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:58.798285 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:58.798283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 412
I20260928 23:16:58.798379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:58.798393 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [term 1 LEADER]: Becoming Leader. State: Replica: 53b82f2e5e5444d998a18197ca9fc605, State: Running, Role: LEADER
I20260928 23:16:58.798462 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b82f2e5e5444d998a18197ca9fc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35111 } }
I20260928 23:16:58.798516 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:58.799198 392 external_mini_cluster.cc:1428] Setting key f4684c213219c3ef40c250d3d3fa6e81
I20260928 23:16:58.799693 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53b82f2e5e5444d998a18197ca9fc605. Latest consensus state: current_term: 1 leader_uuid: "53b82f2e5e5444d998a18197ca9fc605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b82f2e5e5444d998a18197ca9fc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35111 } } }
I20260928 23:16:58.799782 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:58.800028 498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:58.800081 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53b82f2e5e5444d998a18197ca9fc605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b82f2e5e5444d998a18197ca9fc605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35111 } } }
I20260928 23:16:58.800220 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:58.800679 498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:58.802131 498 catalog_manager.cc:1347] Generated new cluster ID: dc4b6cc452ac4998a4a2e9df4c3f52bb
I20260928 23:16:58.802182 498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:16:58 dist-test-slave-djkr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:16:58.812536 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:58.800268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.0.98.62:35111)
0928 23:16:58.800418 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:58.800427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:58.800646 (+ 219us) server_negotiation.cc:407] Connection header received
0928 23:16:58.802556 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:58.802562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:58.802605 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:58.802709 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:58.804151 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:58.805021 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:58.806022 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:58.806407 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:58.806497 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:58.809448 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:58.809451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:58.809451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:58.809460 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:58.810905 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:58.811271 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:58.811274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:58.811275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:58.811362 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:58.811609 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:58.811610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:58.811611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:58.811821 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:58.811959 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:58.812215 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 23:16:58.812337 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260928 23:16:58.819402 498 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:58.819916 498 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:58.824221 498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53b82f2e5e5444d998a18197ca9fc605: Generated new TSK 0
I20260928 23:16:58.824371 498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:16:58.850145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35111
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:58.949433 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:58.949621 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:58.949643 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:58.950765 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:58.950803 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:58.950829 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:16:58.952241 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:58.952442 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:58.952668 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:58.957337 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:58.957314 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:58.957334 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:58.957479 506 server_base.cc:1034] running on GCE node
I20260928 23:16:58.957711 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:58.957947 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:58.959089 506 hybrid_clock.cc:648] HybridClock initialized: now 1790637418959078 us; error 31 us; skew 500 ppm
Sep 28 23:16:58 dist-test-slave-djkr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:58.961445 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:16:58.961864 506 webserver.cc:460] Webserver started at http://127.0.98.1:44151/ using document root <none> and password file <none>
I20260928 23:16:58.962071 506 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:58.962114 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:58.962221 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:58.963239 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "d944df17b48d45d5a86dad0e917401ed"
format_stamp: "Formatted at 2026-09-28 23:16:58 on dist-test-slave-djkr"
server_key: "43f21fe79be589fb4e39071b2ddaffc1"
server_key_iv: "a45570a801dbf66646dbdc655827298c"
server_key_version: "encryptionkey@0"
I20260928 23:16:58.963580 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d944df17b48d45d5a86dad0e917401ed"
format_stamp: "Formatted at 2026-09-28 23:16:58 on dist-test-slave-djkr"
server_key: "43f21fe79be589fb4e39071b2ddaffc1"
server_key_iv: "a45570a801dbf66646dbdc655827298c"
server_key_version: "encryptionkey@0"
I20260928 23:16:58.964777 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:16:58.965477 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:58.965662 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:16:58.965740 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "d944df17b48d45d5a86dad0e917401ed"
format_stamp: "Formatted at 2026-09-28 23:16:58 on dist-test-slave-djkr"
server_key: "43f21fe79be589fb4e39071b2ddaffc1"
server_key_iv: "a45570a801dbf66646dbdc655827298c"
server_key_version: "encryptionkey@0"
I20260928 23:16:58.965801 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:58.984081 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:58.985754 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:58.985862 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:58.986020 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:16:58.986248 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:16:58.986276 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:58.986299 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:16:58.986310 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:58.991597 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35965
I20260928 23:16:58.991660 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35965 every 8 connection(s)
I20260928 23:16:58.991915 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:16:58.994776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 506
I20260928 23:16:58.994860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:16:58.994997 392 external_mini_cluster.cc:1428] Setting key 69d835cdb1cfa3d164132d3107f0d5eb
Sep 28 23:16:58 dist-test-slave-djkr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:16:59.001041 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:58.992532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35185 (local address 127.0.98.62:35111)
0928 23:16:58.992586 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:58.992588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:58.992743 (+ 155us) server_negotiation.cc:407] Connection header received
0928 23:16:58.993193 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:58.993194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:58.993206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:58.993253 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:58.994082 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:58.994583 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:58.995799 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:58.995997 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:58.996050 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:58.998178 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:58.998179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:58.998180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:58.998187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:58.999644 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:59.000055 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:59.000056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:59.000056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:59.000114 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:59.000391 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:59.000392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:59.000393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:59.000511 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:59.000565 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:59.000893 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 23:16:59.000966 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:16:59.001230 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:58.992494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35111 (local address 127.0.98.1:35185)
0928 23:16:58.992635 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:58.992672 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:58.993119 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:58.993315 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:58.993318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:58.993835 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:58.994007 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:58.994011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:58.994650 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:58.994651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:58.995711 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:58.995714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:58.996066 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:58.996067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:58.996203 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:58.996598 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:58.996606 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:58.998089 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:58.999734 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:58.999740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:58.999742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:58.999984 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:59.000195 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:59.000196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:59.000196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:59.000327 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:59.000571 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:59.000577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:59.000731 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 23:16:59.000859 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 23:16:59.001019 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260928 23:16:59.001658 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35111
I20260928 23:16:59.001754 635 heartbeater.cc:464] Registering TS with master...
I20260928 23:16:59.001941 635 heartbeater.cc:511] Master 127.0.98.62:35111 requested a full tablet report, sending...
I20260928 23:16:59.002460 444 ts_manager.cc:194] Registered new tserver with Master: d944df17b48d45d5a86dad0e917401ed (127.0.98.1:35965)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:16:59.003129 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:16:59.015913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35111
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:59.092592 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:59.092770 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:59.092796 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:59.093868 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:59.093912 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:59.093937 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:16:59.095312 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:59.095523 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:59.095736 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:59.100241 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:59.100242 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:59.100303 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:59.100477 640 server_base.cc:1034] running on GCE node
I20260928 23:16:59.100651 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:59.100862 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:59.102006 640 hybrid_clock.cc:648] HybridClock initialized: now 1790637419101987 us; error 29 us; skew 500 ppm
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:59.104326 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:16:59.104694 640 webserver.cc:460] Webserver started at http://127.0.98.2:43409/ using document root <none> and password file <none>
I20260928 23:16:59.104883 640 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:59.104928 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:59.105047 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:59.105997 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "7116c96b5e394d0b8e9bf02214322103"
format_stamp: "Formatted at 2026-09-28 23:16:59 on dist-test-slave-djkr"
server_key: "4ff1a083c88fcb5db818e842126cca08"
server_key_iv: "b1d8ffc17b66d7e88c996072cfb82d63"
server_key_version: "encryptionkey@0"
I20260928 23:16:59.106314 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7116c96b5e394d0b8e9bf02214322103"
format_stamp: "Formatted at 2026-09-28 23:16:59 on dist-test-slave-djkr"
server_key: "4ff1a083c88fcb5db818e842126cca08"
server_key_iv: "b1d8ffc17b66d7e88c996072cfb82d63"
server_key_version: "encryptionkey@0"
I20260928 23:16:59.107457 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:16:59.108081 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:59.108259 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:59.108335 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "7116c96b5e394d0b8e9bf02214322103"
format_stamp: "Formatted at 2026-09-28 23:16:59 on dist-test-slave-djkr"
server_key: "4ff1a083c88fcb5db818e842126cca08"
server_key_iv: "b1d8ffc17b66d7e88c996072cfb82d63"
server_key_version: "encryptionkey@0"
I20260928 23:16:59.108395 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:59.114632 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:59.116495 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:59.116616 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:59.116793 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:16:59.117038 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:16:59.117067 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:59.117092 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:16:59.117117 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:59.123062 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42399
I20260928 23:16:59.123103 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42399 every 8 connection(s)
I20260928 23:16:59.123399 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:16:59.129907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 640
I20260928 23:16:59.129997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:16:59.130164 392 external_mini_cluster.cc:1428] Setting key 65db8aa9e2a5e1779232c2683846e022
I20260928 23:16:59.132479 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:59.124074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33181 (local address 127.0.98.62:35111)
0928 23:16:59.124144 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:59.124146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:59.124363 (+ 217us) server_negotiation.cc:407] Connection header received
0928 23:16:59.124783 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:59.124784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:59.124794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:59.124848 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:59.125736 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.126236 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.127194 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.127395 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.127450 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:59.129596 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:59.129597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:59.129597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:59.129603 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:59.131111 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:59.131527 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:59.131528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:59.131528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:59.131604 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:59.131868 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:59.131868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:59.131868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:59.131960 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:59.132003 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:59.132306 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 23:16:59.132389 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:16:59.132597 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:59.124062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35111 (local address 127.0.98.2:33181)
0928 23:16:59.124204 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:59.124261 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:59.124707 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:59.124904 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:59.124907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:59.125467 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:59.125654 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:59.125659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.126313 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.126314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:59.127115 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:59.127117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.127496 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.127497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:59.127663 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:59.128150 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:59.128159 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:59.129508 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:59.131185 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:59.131191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:59.131193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:59.131464 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:59.131657 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:59.131657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:59.131658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:59.131804 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:59.132009 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:59.132013 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:59.132169 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 23:16:59.132283 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:16:59.132423 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":84,"threads_started":1}
I20260928 23:16:59.133024 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35111
I20260928 23:16:59.133103 769 heartbeater.cc:464] Registering TS with master...
I20260928 23:16:59.133289 769 heartbeater.cc:511] Master 127.0.98.62:35111 requested a full tablet report, sending...
I20260928 23:16:59.133625 444 ts_manager.cc:194] Registered new tserver with Master: 7116c96b5e394d0b8e9bf02214322103 (127.0.98.2:42399)
I20260928 23:16:59.133965 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33181
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:16:59.152107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35111
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:59.230448 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:59.230648 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:59.230676 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:59.231794 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:59.231832 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:59.231866 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:16:59.233238 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:59.233469 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:59.233660 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:59.238559 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:59.238660 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:59.238571 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:59.238670 774 server_base.cc:1034] running on GCE node
I20260928 23:16:59.238942 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:59.239132 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:59.240284 774 hybrid_clock.cc:648] HybridClock initialized: now 1790637419240267 us; error 30 us; skew 500 ppm
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:59.242585 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:16:59.242944 774 webserver.cc:460] Webserver started at http://127.0.98.3:40237/ using document root <none> and password file <none>
I20260928 23:16:59.243114 774 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:59.243152 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:59.243235 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:59.244220 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ddc1bb1f1094426aad4c30d9256795c"
format_stamp: "Formatted at 2026-09-28 23:16:59 on dist-test-slave-djkr"
server_key: "2a49fa786ffd57e07cde6bfa216c8a71"
server_key_iv: "daa7ed3c7347ca563a2b3a7f718ffbc5"
server_key_version: "encryptionkey@0"
I20260928 23:16:59.244520 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ddc1bb1f1094426aad4c30d9256795c"
format_stamp: "Formatted at 2026-09-28 23:16:59 on dist-test-slave-djkr"
server_key: "2a49fa786ffd57e07cde6bfa216c8a71"
server_key_iv: "daa7ed3c7347ca563a2b3a7f718ffbc5"
server_key_version: "encryptionkey@0"
I20260928 23:16:59.245654 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:16:59.246352 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:59.246495 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:59.246565 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "3ddc1bb1f1094426aad4c30d9256795c"
format_stamp: "Formatted at 2026-09-28 23:16:59 on dist-test-slave-djkr"
server_key: "2a49fa786ffd57e07cde6bfa216c8a71"
server_key_iv: "daa7ed3c7347ca563a2b3a7f718ffbc5"
server_key_version: "encryptionkey@0"
I20260928 23:16:59.246626 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:59.269155 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:59.270871 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:59.270988 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:59.271189 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:16:59.271472 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:16:59.271510 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:59.271538 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:16:59.271588 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:59.277009 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42837
I20260928 23:16:59.277055 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42837 every 8 connection(s)
I20260928 23:16:59.277428 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:16:59.285589 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:59.278040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60915 (local address 127.0.98.62:35111)
0928 23:16:59.278081 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:59.278083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:59.278290 (+ 207us) server_negotiation.cc:407] Connection header received
0928 23:16:59.278767 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:59.278769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:59.278780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:59.278839 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:59.279814 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.280293 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.281227 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.281406 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.281459 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:59.283362 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:59.283364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:59.283364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:59.283372 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:59.284441 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:59.284763 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:59.284765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:59.284765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:59.284825 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:59.285022 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:59.285023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:59.285023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:59.285123 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:59.285176 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:59.285429 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 23:16:59.285502 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:16:59.285739 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:59.278027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35111 (local address 127.0.98.3:60915)
0928 23:16:59.278176 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:59.278209 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:59.278674 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:59.278917 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:59.278921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:59.279463 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:59.279736 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:59.279743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.280368 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.280369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:59.281147 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:59.281149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.281468 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.281469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:59.281583 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:59.281987 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:59.281994 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:59.283257 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:59.284506 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:59.284512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:59.284514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:59.284700 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:59.284876 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:59.284877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:59.284877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:59.284970 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:59.285174 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:59.285179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:59.285317 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 23:16:59.285403 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:16:59.285543 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260928 23:16:59.286151 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35111
I20260928 23:16:59.286253 903 heartbeater.cc:464] Registering TS with master...
I20260928 23:16:59.286355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 774
I20260928 23:16:59.286420 903 heartbeater.cc:511] Master 127.0.98.62:35111 requested a full tablet report, sending...
I20260928 23:16:59.286438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:16:59.286589 392 external_mini_cluster.cc:1428] Setting key 0063d05245d77dca56f441d00b46a05b
I20260928 23:16:59.286789 444 ts_manager.cc:194] Registered new tserver with Master: 3ddc1bb1f1094426aad4c30d9256795c (127.0.98.3:42837)
I20260928 23:16:59.287238 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60915
I20260928 23:16:59.287622 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637419, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637419, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:16:59.308785 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:59.301799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.0.98.1:35965)
0928 23:16:59.301986 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:59.301991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:59.302003 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:16:59.302028 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:59.302030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:59.302044 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:59.302108 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:59.302623 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.303339 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.304075 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.304385 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.304439 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:59.306472 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:59.306479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:59.306480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:59.306486 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:59.307860 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:59.308157 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:59.308160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:59.308160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:59.308222 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:59.308351 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:59.308351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:59.308352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:59.308496 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:59.308597 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:59.308649 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:16:59.308707 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
W20260928 23:16:59.309127 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41080
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:16:59.329257 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:59.323059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.0.98.1:35965)
0928 23:16:59.323111 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:59.323114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:59.323172 (+ 58us) server_negotiation.cc:407] Connection header received
0928 23:16:59.323229 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:59.323230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:59.323240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:59.323285 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:59.323868 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.324416 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.325142 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:59.325365 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:59.325423 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:59.327436 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:59.327437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:59.327438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:59.327446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:59.328490 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:59.328758 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:59.328758 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:59.328759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:59.328812 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:59.328928 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:59.328928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:59.328928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:59.329002 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:59.329079 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:59.329132 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 23:16:59.329191 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:16:59.329681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 506
I20260928 23:16:59.334453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 640
I20260928 23:16:59.339211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 774
I20260928 23:16:59.342950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 412
2026-09-28T23:16:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1713 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[917](info): set up 2 sockets
Sep 28 23:16:59 dist-test-slave-djkr krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 23:17:00.286772 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:00 dist-test-slave-djkr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:00.392606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45417
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:00.470953 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:00.471153 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:00.471174 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:00.472373 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:00.472419 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:00.472431 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:00.472443 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:00.472453 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:00.473917 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:00.474133 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:00.474351 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:00.478858 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:00.478870 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:00.478978 930 server_base.cc:1034] running on GCE node
W20260928 23:17:00.478870 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:00.479326 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:00.479650 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:00.480785 930 hybrid_clock.cc:648] HybridClock initialized: now 1790637420480776 us; error 34 us; skew 500 ppm
Sep 28 23:17:00 dist-test-slave-djkr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:00.483073 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:00.483507 930 webserver.cc:460] Webserver started at http://127.0.98.62:40497/ using document root <none> and password file <none>
I20260928 23:17:00.483738 930 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:00.483786 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:00.483903 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:00.484895 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "fedbde8f5a26c602b95874fda6fbdd54"
server_key_iv: "11016e971284b2cf1922a01f0b1fb8bd"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.485219 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "fedbde8f5a26c602b95874fda6fbdd54"
server_key_iv: "11016e971284b2cf1922a01f0b1fb8bd"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.486438 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:00.487124 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.487270 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:00.487358 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "fedbde8f5a26c602b95874fda6fbdd54"
server_key_iv: "11016e971284b2cf1922a01f0b1fb8bd"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.487423 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:00.497721 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:00.499526 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:00.499684 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:00.503158 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45417
I20260928 23:17:00.503206 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45417 every 8 connection(s)
I20260928 23:17:00.503516 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:00.504305 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:00.506243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 930
I20260928 23:17:00.506336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:00.506472 392 external_mini_cluster.cc:1428] Setting key d4f1f4a5700cec2893725ed78cd1f77e
I20260928 23:17:00.506518 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55: Bootstrap starting.
I20260928 23:17:00.507156 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:00.507426 998 log.cc:824] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:00.508091 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55: No bootstrap required, opened a new log
I20260928 23:17:00.510142 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } }
I20260928 23:17:00.510318 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:00.510366 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d8a6e36e3a4b1e856f4ec85b15ad55, State: Initialized, Role: FOLLOWER
I20260928 23:17:00.510483 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } }
I20260928 23:17:00.510557 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:00.510594 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:00.510633 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:00.511162 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } }
I20260928 23:17:00.511269 998 leader_election.cc:302] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1d8a6e36e3a4b1e856f4ec85b15ad55; no voters:
I20260928 23:17:00.511425 998 leader_election.cc:288] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:00.511472 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:00.511600 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [term 1 LEADER]: Becoming Leader. State: Replica: d1d8a6e36e3a4b1e856f4ec85b15ad55, State: Running, Role: LEADER
I20260928 23:17:00.511677 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } }
I20260928 23:17:00.511706 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:00.511991 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } } }
I20260928 23:17:00.512048 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:00.512153 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1d8a6e36e3a4b1e856f4ec85b15ad55. Latest consensus state: current_term: 1 leader_uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d8a6e36e3a4b1e856f4ec85b15ad55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } } }
I20260928 23:17:00.512194 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55 [sys.catalog]: This master's current role is: LEADER
Sep 28 23:17:00 dist-test-slave-djkr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260928 23:17:00.513388 1019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:17:00.513447 1019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:17:00.513489 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:00.513640 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:00.515305 1008 catalog_manager.cc:1347] Generated new cluster ID: 8768971c731244bcac9da13ddb1b5788
I20260928 23:17:00.515353 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:00.517143 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.506952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.0.98.62:45417)
0928 23:17:00.507102 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:00.507111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:00.507135 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:17:00.507781 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:00.507790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:00.507841 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:00.507961 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:00.508684 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.509574 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.510578 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.511044 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.511130 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.513732 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:00.513736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:00.513736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:00.513749 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:00.515521 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:00.515940 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:00.515943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:00.515943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:00.516056 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:00.516244 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:00.516245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:00.516245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:00.516548 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:00.516698 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:00.516775 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 23:17:00.516930 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260928 23:17:00.520767 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:00.521443 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:00.525553 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1d8a6e36e3a4b1e856f4ec85b15ad55: Generated new TSK 0
I20260928 23:17:00.525715 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:17:00.549705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:00.627296 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:00.627481 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:00.627504 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:00.628643 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:00.628681 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:00.628706 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:00.630052 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:00.630244 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:00.630448 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:00.635275 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:00.635391 1024 server_base.cc:1034] running on GCE node
W20260928 23:17:00.635313 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:00.635268 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:00.635730 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:00.635954 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:00.637095 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790637420637079 us; error 27 us; skew 500 ppm
Sep 28 23:17:00 dist-test-slave-djkr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:00.639343 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:17:00.639750 1024 webserver.cc:460] Webserver started at http://127.0.98.1:40923/ using document root <none> and password file <none>
I20260928 23:17:00.639919 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:00.639956 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:00.640039 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:00.640975 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "71c1ec0d0bba4969a959fabd60271e6a"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "bdd4b803183a8a3b5da6c3a09bf9b04a"
server_key_iv: "7fe63086ad234a80df043a3016b5f752"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.641254 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71c1ec0d0bba4969a959fabd60271e6a"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "bdd4b803183a8a3b5da6c3a09bf9b04a"
server_key_iv: "7fe63086ad234a80df043a3016b5f752"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.642329 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:17:00.642927 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.643050 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.643121 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "71c1ec0d0bba4969a959fabd60271e6a"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "bdd4b803183a8a3b5da6c3a09bf9b04a"
server_key_iv: "7fe63086ad234a80df043a3016b5f752"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.643182 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:00.661020 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:00.663049 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:00.663160 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:00.663357 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:00.663650 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:00.663683 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.663704 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:00.663728 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.669006 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34315
I20260928 23:17:00.669062 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34315 every 8 connection(s)
I20260928 23:17:00.669327 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:00.673681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 1024
I20260928 23:17:00.673756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:00.673890 392 external_mini_cluster.cc:1428] Setting key 97fe92293210a011778ce98ab1d39a60
Sep 28 23:17:00 dist-test-slave-djkr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:00.678226 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.669967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35269 (local address 127.0.98.62:45417)
0928 23:17:00.670013 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:00.670016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:00.670192 (+ 176us) server_negotiation.cc:407] Connection header received
0928 23:17:00.670656 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:00.670657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:00.670668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:00.670720 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:00.671576 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.672078 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.673021 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.673210 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.673256 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.675580 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:00.675581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:00.675581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:00.675589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:00.676942 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:00.677383 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:00.677384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:00.677384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:00.677435 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:00.677667 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:00.677668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:00.677668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:00.677756 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:00.677800 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:00.678101 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 23:17:00.678166 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:17:00.678468 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.669943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45417 (local address 127.0.98.1:35269)
0928 23:17:00.670083 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:00.670116 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:00.670557 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:00.670793 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:00.670796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:00.671296 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:00.671475 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.671479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.672168 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.672169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.672955 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.672958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.673269 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.673270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.673359 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.673815 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:00.673825 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:00.675464 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:00.677016 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:00.677022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:00.677024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:00.677281 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:00.677485 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:00.677486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:00.677486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:00.677613 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:00.677810 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:00.677815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:00.677975 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 23:17:00.678088 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 23:17:00.678244 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260928 23:17:00.678916 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45417
I20260928 23:17:00.678997 1153 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:00.679149 1153 heartbeater.cc:511] Master 127.0.98.62:45417 requested a full tablet report, sending...
I20260928 23:17:00.679625 962 ts_manager.cc:194] Registered new tserver with Master: 71c1ec0d0bba4969a959fabd60271e6a (127.0.98.1:34315)
I20260928 23:17:00.680171 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35269
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:17:00.695796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:00.771890 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:00.772066 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:00.772096 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:00.773211 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:00.773249 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:00.773283 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:00.774629 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:00.774887 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:00.775081 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:00.779539 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:00.779584 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:00.779569 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:00.779585 1158 server_base.cc:1034] running on GCE node
I20260928 23:17:00.779906 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:00.780085 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:00.781224 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790637420781211 us; error 27 us; skew 500 ppm
Sep 28 23:17:00 dist-test-slave-djkr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:00.783421 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:17:00.783815 1158 webserver.cc:460] Webserver started at http://127.0.98.2:40633/ using document root <none> and password file <none>
I20260928 23:17:00.784008 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:00.784051 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:00.784163 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:00.785076 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "4b622c817b0cd65ee6b27574051c89b7"
server_key_iv: "4d3380e5c23d4716cbe82023ff37fd8d"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.785392 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "4b622c817b0cd65ee6b27574051c89b7"
server_key_iv: "4d3380e5c23d4716cbe82023ff37fd8d"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.786510 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:00.787101 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.787238 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:00.787340 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "4b622c817b0cd65ee6b27574051c89b7"
server_key_iv: "4d3380e5c23d4716cbe82023ff37fd8d"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.787395 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:00.796337 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:00.797842 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:00.797947 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:00.798123 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:00.798372 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:00.798403 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.798427 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:00.798450 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.803607 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40435
I20260928 23:17:00.803653 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40435 every 8 connection(s)
I20260928 23:17:00.803925 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:17:00 dist-test-slave-djkr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:00.809216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 1158
I20260928 23:17:00.809293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:00.809430 392 external_mini_cluster.cc:1428] Setting key 614806ab5126fc74cc985f5e2f36a39d
I20260928 23:17:00.811789 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.804520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50301 (local address 127.0.98.62:45417)
0928 23:17:00.804587 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:00.804589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:00.804717 (+ 128us) server_negotiation.cc:407] Connection header received
0928 23:17:00.805122 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:00.805123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:00.805132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:00.805177 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:00.805997 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.806471 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.807340 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.807511 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.807576 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.809399 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:00.809401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:00.809402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:00.809409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:00.810591 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:00.810928 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:00.810929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:00.810929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:00.810985 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:00.811182 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:00.811183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:00.811183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:00.811288 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:00.811341 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:00.811625 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 23:17:00.811699 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:17:00.811908 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.804493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45417 (local address 127.0.98.2:50301)
0928 23:17:00.804619 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:00.804650 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:00.805055 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:00.805242 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:00.805245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:00.805748 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:00.805932 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.805936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.806525 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.806526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.807286 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.807288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.807591 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.807592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.807692 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.808092 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:00.808100 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:00.809299 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:00.810661 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:00.810667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:00.810669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:00.810859 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:00.811035 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:00.811036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:00.811036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:00.811130 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:00.811341 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:00.811346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:00.811468 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 23:17:00.811567 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 23:17:00.811706 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":75,"threads_started":1}
I20260928 23:17:00.812266 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45417
I20260928 23:17:00.812366 1287 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:00.812520 1287 heartbeater.cc:511] Master 127.0.98.62:45417 requested a full tablet report, sending...
I20260928 23:17:00.812870 962 ts_manager.cc:194] Registered new tserver with Master: 1c0ca08bea2d444b8c180ee1f9e7cbff (127.0.98.2:40435)
I20260928 23:17:00.813354 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50301
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:17:00.829816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:00.905161 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:00.905371 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:00.905393 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:00.906538 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:00.906574 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:00.906598 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:00.907994 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:00.908195 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:00.908393 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:00.912982 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:00.913093 1292 server_base.cc:1034] running on GCE node
W20260928 23:17:00.913002 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:00.912982 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:00.913352 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:00.913543 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:00.914691 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790637420914678 us; error 35 us; skew 500 ppm
Sep 28 23:17:00 dist-test-slave-djkr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:00.916985 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:17:00.917403 1292 webserver.cc:460] Webserver started at http://127.0.98.3:34663/ using document root <none> and password file <none>
I20260928 23:17:00.917600 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:00.917644 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:00.917763 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:00.918722 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "ddcee01b32aa4863bde2aa4a72e1f993"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "3e2c5feba39c9a2d497bd952bbe3c53e"
server_key_iv: "dd736a689130d513eb005f95d3aed923"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.919047 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ddcee01b32aa4863bde2aa4a72e1f993"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "3e2c5feba39c9a2d497bd952bbe3c53e"
server_key_iv: "dd736a689130d513eb005f95d3aed923"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.920249 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:17:00.920861 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.921000 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:00.921067 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "ddcee01b32aa4863bde2aa4a72e1f993"
format_stamp: "Formatted at 2026-09-28 23:17:00 on dist-test-slave-djkr"
server_key: "3e2c5feba39c9a2d497bd952bbe3c53e"
server_key_iv: "dd736a689130d513eb005f95d3aed923"
server_key_version: "encryptionkey@0"
I20260928 23:17:00.921123 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:00.935972 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:00.937603 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:00.937726 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:00.937918 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:00.938179 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:00.938210 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.938237 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:00.938261 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:00.943420 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36833
I20260928 23:17:00.943454 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36833 every 8 connection(s)
I20260928 23:17:00.943758 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:17:00 dist-test-slave-djkr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:00.952289 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.944398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59823 (local address 127.0.98.62:45417)
0928 23:17:00.944449 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:00.944451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:00.944645 (+ 194us) server_negotiation.cc:407] Connection header received
0928 23:17:00.945141 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:00.945142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:00.945153 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:00.945200 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:00.946183 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.946684 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.947724 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.947914 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.947968 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.949889 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:00.949891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:00.949891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:00.949898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:00.951077 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:00.951431 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:00.951432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:00.951433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:00.951483 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:00.951723 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:00.951724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:00.951724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:00.951828 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:00.951883 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:00.952126 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 23:17:00.952199 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:17:00.952462 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.944368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45417 (local address 127.0.98.3:59823)
0928 23:17:00.944529 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:00.944567 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:00.945048 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:00.945319 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:00.945322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:00.945906 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:00.946093 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.946098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.946771 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.946772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.947631 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.947634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.947980 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.947980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.948099 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.948561 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:00.948569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:00.949806 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:00.951148 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:00.951154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:00.951156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:00.951368 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:00.951534 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:00.951535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:00.951535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:00.951661 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:00.951881 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:00.951888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:00.952014 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 23:17:00.952099 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:17:00.952254 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260928 23:17:00.952924 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45417
I20260928 23:17:00.953014 1421 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:00.953187 1421 heartbeater.cc:511] Master 127.0.98.62:45417 requested a full tablet report, sending...
I20260928 23:17:00.953583 962 ts_manager.cc:194] Registered new tserver with Master: ddcee01b32aa4863bde2aa4a72e1f993 (127.0.98.3:36833)
I20260928 23:17:00.953598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 1292
I20260928 23:17:00.953670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:00.953817 392 external_mini_cluster.cc:1428] Setting key 140675c189b6b0076351f37891c9ef14
I20260928 23:17:00.953980 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59823
I20260928 23:17:00.954581 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:17:00.961319 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.956100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.0.98.62:45417)
0928 23:17:00.956134 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:00.956136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:00.956261 (+ 125us) server_negotiation.cc:407] Connection header received
0928 23:17:00.956324 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:00.956324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:00.956333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:00.956377 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:00.956880 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.957334 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.958010 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.958187 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.958237 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.959225 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:00.959226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:00.959226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:00.959232 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:00.960385 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:00.960663 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:00.960664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:00.960664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:00.960711 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:00.960865 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:00.960866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:00.960866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:00.960990 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:00.961073 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:00.961168 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 23:17:00.961238 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:17:00.963279 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:00.964762 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:00.970533 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.966754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36833 (local address 127.0.0.1:34188)
0928 23:17:00.966907 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:00.966915 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:00.966952 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:00.967416 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:00.967416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:00.967418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:00.967449 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.967450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.968670 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.968671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.969394 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.969396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.970241 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.970242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.970323 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.970337 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:17:00.970404 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:17:00.970456 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260928 23:17:00.970534 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.966815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.0.98.3:36833)
0928 23:17:00.967280 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:00.967284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:00.967292 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:00.967324 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:00.967326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:00.967340 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:00.967409 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:00.967516 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.968595 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.969470 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.970160 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.970230 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.970285 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:00.970425 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:17:00.970461 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":73,"threads_started":1}
I20260928 23:17:00.970618 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.966663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.0.98.2:40435)
0928 23:17:00.966991 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:00.966996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:00.967006 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:00.967036 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:00.967038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:00.967049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:00.967121 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:00.967246 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.968166 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.969709 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.970188 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.970249 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.970301 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:00.970503 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 23:17:00.970546 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":74,"threads_started":1}
I20260928 23:17:00.970932 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.966733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.0.98.1:34315)
0928 23:17:00.967045 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:00.967049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:00.967056 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:00.967086 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:00.967087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:00.967098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:00.967158 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:00.967286 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.968387 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.969601 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.970268 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.970323 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.970363 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:00.970814 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 23:17:00.970860 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":78,"threads_started":1}
I20260928 23:17:00.971056 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.966572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34315 (local address 127.0.0.1:42936)
0928 23:17:00.966861 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:00.966870 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:00.966917 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:00.967165 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:00.967166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:00.967168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:00.967212 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.967214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.968477 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.968478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.969523 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.969525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.970403 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.970404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.970466 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.970470 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:00.970518 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:17:00.970558 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260928 23:17:00.971163 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:00.966571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40435 (local address 127.0.0.1:58672)
0928 23:17:00.966695 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:00.966732 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:00.966872 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:00.967125 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:00.967128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:00.967130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:00.967186 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.967188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.968247 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.968248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.969637 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:00.969638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:00.970331 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:00.970331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:00.970379 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:00.970382 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:00.970424 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:17:00.970456 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":22,"thread_start_us":55,"threads_started":1}
I20260928 23:17:00.971385 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 7125c1de6ddc401d91b8f5ba4d2140ba (DEFAULT_TABLE table=test-table [id=b587e897a10641ee9184351bf7c59960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:00.971700 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7125c1de6ddc401d91b8f5ba4d2140ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:00.971699 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 7125c1de6ddc401d91b8f5ba4d2140ba (DEFAULT_TABLE table=test-table [id=b587e897a10641ee9184351bf7c59960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:00.972005 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7125c1de6ddc401d91b8f5ba4d2140ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:00.974026 1435 tablet_bootstrap.cc:492] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993: Bootstrap starting.
I20260928 23:17:00.974248 1436 tablet_bootstrap.cc:492] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a: Bootstrap starting.
I20260928 23:17:00.974669 1435 tablet_bootstrap.cc:654] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:00.974670 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 7125c1de6ddc401d91b8f5ba4d2140ba (DEFAULT_TABLE table=test-table [id=b587e897a10641ee9184351bf7c59960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:00.974797 1436 tablet_bootstrap.cc:654] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:00.975001 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7125c1de6ddc401d91b8f5ba4d2140ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:00.975206 1435 log.cc:824] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:00.975250 1436 log.cc:824] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:00.975924 1436 tablet_bootstrap.cc:492] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a: No bootstrap required, opened a new log
I20260928 23:17:00.975924 1435 tablet_bootstrap.cc:492] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993: No bootstrap required, opened a new log
I20260928 23:17:00.975989 1436 ts_tablet_manager.cc:1397] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:00.975991 1435 ts_tablet_manager.cc:1397] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:00.977465 1439 tablet_bootstrap.cc:492] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff: Bootstrap starting.
I20260928 23:17:00.977795 1435 raft_consensus.cc:348] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40435 } } peers { permanent_uuid: "71c1ec0d0bba4969a959fabd60271e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34315 } } peers { permanent_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260928 23:17:00.977795 1436 raft_consensus.cc:348] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40435 } } peers { permanent_uuid: "71c1ec0d0bba4969a959fabd60271e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34315 } } peers { permanent_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260928 23:17:00.977970 1435 raft_consensus.cc:374] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:00.977970 1436 raft_consensus.cc:374] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:00.978005 1436 raft_consensus.cc:729] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c1ec0d0bba4969a959fabd60271e6a, State: Initialized, Role: FOLLOWER
I20260928 23:17:00.978005 1435 raft_consensus.cc:729] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddcee01b32aa4863bde2aa4a72e1f993, State: Initialized, Role: FOLLOWER
I20260928 23:17:00.978080 1439 tablet_bootstrap.cc:654] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:00.978118 1435 consensus_queue.cc:260] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40435 } } peers { permanent_uuid: "71c1ec0d0bba4969a959fabd60271e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34315 } } peers { permanent_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260928 23:17:00.978129 1436 consensus_queue.cc:260] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40435 } } peers { permanent_uuid: "71c1ec0d0bba4969a959fabd60271e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34315 } } peers { permanent_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260928 23:17:00.978392 1435 ts_tablet_manager.cc:1428] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:17:00.978386 1436 ts_tablet_manager.cc:1428] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:17:00.978416 1421 heartbeater.cc:503] Master 127.0.98.62:45417 was elected leader, sending a full tablet report...
I20260928 23:17:00.978502 1153 heartbeater.cc:503] Master 127.0.98.62:45417 was elected leader, sending a full tablet report...
I20260928 23:17:00.979007 1439 log.cc:824] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:00.979740 1439 tablet_bootstrap.cc:492] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff: No bootstrap required, opened a new log
I20260928 23:17:00.979791 1439 ts_tablet_manager.cc:1397] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:00.981151 1439 raft_consensus.cc:348] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40435 } } peers { permanent_uuid: "71c1ec0d0bba4969a959fabd60271e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34315 } } peers { permanent_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260928 23:17:00.981274 1439 raft_consensus.cc:374] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:00.981294 1439 raft_consensus.cc:729] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0ca08bea2d444b8c180ee1f9e7cbff, State: Initialized, Role: FOLLOWER
I20260928 23:17:00.981396 1439 consensus_queue.cc:260] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40435 } } peers { permanent_uuid: "71c1ec0d0bba4969a959fabd60271e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34315 } } peers { permanent_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260928 23:17:00.981601 1439 ts_tablet_manager.cc:1428] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:17:00.981635 1287 heartbeater.cc:503] Master 127.0.98.62:45417 was elected leader, sending a full tablet report...
W20260928 23:17:01.054426 1288 tablet.cc:2369] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:17:01.169947 1154 tablet.cc:2369] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:17:01.194275 1422 tablet.cc:2369] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:01.345636 1441 raft_consensus.cc:482] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:01.345820 1441 raft_consensus.cc:504] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40435 } } peers { permanent_uuid: "71c1ec0d0bba4969a959fabd60271e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34315 } } peers { permanent_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260928 23:17:01.346169 1441 leader_election.cc:288] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c0ca08bea2d444b8c180ee1f9e7cbff (127.0.98.2:40435), 71c1ec0d0bba4969a959fabd60271e6a (127.0.98.1:34315)
I20260928 23:17:01.349319 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:01.346459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53945 (local address 127.0.98.2:40435)
0928 23:17:01.346499 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:01.346501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:01.346514 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:17:01.346579 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:01.346580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:01.346592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:01.346647 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:01.346826 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.347320 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.348419 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.348951 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.349034 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:01.349045 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:01.349212 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 23:17:01.349252 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:17:01.349363 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:01.346456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53333 (local address 127.0.98.1:34315)
0928 23:17:01.346510 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:01.346512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:01.346667 (+ 155us) server_negotiation.cc:407] Connection header received
0928 23:17:01.346704 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:01.346706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:01.346719 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:01.346778 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:01.346914 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.347389 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.348647 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.349036 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.349112 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:01.349123 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:01.349262 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 23:17:01.349304 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:17:01.349356 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:01.346387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40435 (local address 127.0.98.3:53945)
0928 23:17:01.346445 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:01.346457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:01.346503 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:01.346663 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:01.346664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:01.346666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:01.346747 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:01.346749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.347400 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.347400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:01.348230 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:01.348232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.349062 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.349063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:01.349142 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:01.349146 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:01.349196 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:17:01.349232 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 23:17:01.349371 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:01.346394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34315 (local address 127.0.98.3:53333)
0928 23:17:01.346579 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:01.346589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:01.346646 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:01.346786 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:01.346786 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:01.346788 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:01.346854 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:01.346855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.347461 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.347462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:01.348544 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:01.348546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.349131 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.349133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:01.349191 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:01.349195 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:01.349248 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:17:01.349281 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":12,"thread_start_us":107,"threads_started":1}
I20260928 23:17:01.349691 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7125c1de6ddc401d91b8f5ba4d2140ba" candidate_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" is_pre_election: true
I20260928 23:17:01.349756 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7125c1de6ddc401d91b8f5ba4d2140ba" candidate_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c1ec0d0bba4969a959fabd60271e6a" is_pre_election: true
I20260928 23:17:01.349849 1241 raft_consensus.cc:2454] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddcee01b32aa4863bde2aa4a72e1f993 in term 0.
I20260928 23:17:01.349898 1107 raft_consensus.cc:2454] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddcee01b32aa4863bde2aa4a72e1f993 in term 0.
I20260928 23:17:01.350075 1311 leader_election.cc:302] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0ca08bea2d444b8c180ee1f9e7cbff, ddcee01b32aa4863bde2aa4a72e1f993; no voters:
I20260928 23:17:01.350210 1441 raft_consensus.cc:2781] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:01.350273 1441 raft_consensus.cc:482] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:01.350294 1441 raft_consensus.cc:3037] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:01.369321 1441 raft_consensus.cc:504] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40435 } } peers { permanent_uuid: "71c1ec0d0bba4969a959fabd60271e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34315 } } peers { permanent_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260928 23:17:01.369508 1441 leader_election.cc:288] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [CANDIDATE]: Term 1 election: Requested vote from peers 1c0ca08bea2d444b8c180ee1f9e7cbff (127.0.98.2:40435), 71c1ec0d0bba4969a959fabd60271e6a (127.0.98.1:34315)
I20260928 23:17:01.369782 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7125c1de6ddc401d91b8f5ba4d2140ba" candidate_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff"
I20260928 23:17:01.369886 1241 raft_consensus.cc:3037] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:01.369908 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7125c1de6ddc401d91b8f5ba4d2140ba" candidate_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c1ec0d0bba4969a959fabd60271e6a"
I20260928 23:17:01.370024 1107 raft_consensus.cc:3037] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:01.370687 1241 raft_consensus.cc:2454] T 7125c1de6ddc401d91b8f5ba4d2140ba P 1c0ca08bea2d444b8c180ee1f9e7cbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddcee01b32aa4863bde2aa4a72e1f993 in term 1.
I20260928 23:17:01.370850 1311 leader_election.cc:302] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c0ca08bea2d444b8c180ee1f9e7cbff, ddcee01b32aa4863bde2aa4a72e1f993; no voters:
I20260928 23:17:01.370873 1107 raft_consensus.cc:2454] T 7125c1de6ddc401d91b8f5ba4d2140ba P 71c1ec0d0bba4969a959fabd60271e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddcee01b32aa4863bde2aa4a72e1f993 in term 1.
I20260928 23:17:01.370936 1441 raft_consensus.cc:2781] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:01.371096 1441 raft_consensus.cc:686] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [term 1 LEADER]: Becoming Leader. State: Replica: ddcee01b32aa4863bde2aa4a72e1f993, State: Running, Role: LEADER
I20260928 23:17:01.371248 1441 consensus_queue.cc:237] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40435 } } peers { permanent_uuid: "71c1ec0d0bba4969a959fabd60271e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34315 } } peers { permanent_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } }
I20260928 23:17:01.372143 962 catalog_manager.cc:5495] T 7125c1de6ddc401d91b8f5ba4d2140ba P ddcee01b32aa4863bde2aa4a72e1f993 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddcee01b32aa4863bde2aa4a72e1f993 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c0ca08bea2d444b8c180ee1f9e7cbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71c1ec0d0bba4969a959fabd60271e6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddcee01b32aa4863bde2aa4a72e1f993" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36833 } health_report { overall_health: HEALTHY } } }
Sep 28 23:17:01 dist-test-slave-djkr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:17:01.418529 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:01.411449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.0.98.1:34315)
0928 23:17:01.411508 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:01.411510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:01.411628 (+ 118us) server_negotiation.cc:407] Connection header received
0928 23:17:01.411681 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:01.411683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:01.411693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:01.411738 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:01.412258 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.412870 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.413587 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.413760 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.413814 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:01.415966 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:01.415980 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:01.415980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:01.415987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:01.417513 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:01.417832 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:01.417836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:01.417836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:01.417893 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:01.418049 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:01.418050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:01.418050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:01.418186 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:01.418319 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:01.418366 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 23:17:01.418439 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 23:17:01 dist-test-slave-djkr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:17:01 dist-test-slave-djkr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:17:01.455226 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:01.448567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.0.98.1:34315)
0928 23:17:01.448618 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:01.448620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:01.448712 (+ 92us) server_negotiation.cc:407] Connection header received
0928 23:17:01.448798 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:01.448800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:01.448808 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:01.448851 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:01.449422 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.449892 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.450633 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.450827 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.450880 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:01.453073 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:01.453074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:01.453075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:01.453084 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:01.454143 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:01.454452 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:01.454454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:01.454454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:01.454504 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:01.454685 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:01.454686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:01.454686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:01.454795 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:01.454878 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:01.455051 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:17:01.455126 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 23:17:01 dist-test-slave-djkr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:17:01 dist-test-slave-djkr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:17:01.555883 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:01.549199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.0.98.1:34315)
0928 23:17:01.549296 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:01.549297 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:01.549329 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:17:01.549407 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:01.549408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:01.549418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:01.549464 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:01.550021 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.550496 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.551187 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:01.551371 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:01.551425 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:01.553599 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:01.553601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:01.553602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:01.553608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:01.554787 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:01.555183 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:01.555184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:01.555185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:01.555238 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:01.555455 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:01.555455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:01.555456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:01.555576 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:01.555654 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:01.555761 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 23:17:01.555816 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 23:17:01.556324 1067 tablet_service.cc:2908] Rejecting scan request for tablet 7125c1de6ddc401d91b8f5ba4d2140ba: Uninitialized: clean time has not yet been initialized
I20260928 23:17:01.557083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 1024
I20260928 23:17:01.561846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 1158
I20260928 23:17:01.565653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 1292
I20260928 23:17:01.569569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 930
2026-09-28T23:17:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2227 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:03 dist-test-slave-djkr krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:03 dist-test-slave-djkr krb5kdc[1459](info): set up 2 sockets
Sep 28 23:17:03 dist-test-slave-djkr krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260928 23:17:04.270919 392 mini_kdc.cc:121] Time spent starting KDC: real 2.681s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 23:17:04.437171 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.166s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 23:17:04.558971 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.122s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 23:17:04.704200 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.145s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/test-user.keytab.
W20260928 23:17:05.177706 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.473s user 0.000s sys 0.002s
Sep 28 23:17:05 dist-test-slave-djkr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260928 23:17:05.813514 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.617s user 0.000s sys 0.003s
I20260928 23:17:05.813918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34319
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:05.891479 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:05.891724 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:05.891754 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:05.892884 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:05.892922 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:17:05.892939 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:05.892956 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:05.892971 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:05.892994 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:05.894450 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34319
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:05.894678 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:05.894879 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:05.899437 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:05.899462 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:05.899494 1472 server_base.cc:1034] running on GCE node
W20260928 23:17:05.899432 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:05.899890 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:05.900157 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:05.901288 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790637425901269 us; error 33 us; skew 500 ppm
Sep 28 23:17:05 dist-test-slave-djkr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:05.912168 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:05.912601 1472 webserver.cc:460] Webserver started at http://127.0.98.62:39015/ using document root <none> and password file <none>
I20260928 23:17:05.912832 1472 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:05.912882 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:05.913005 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:05.914084 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "8de16dd417064140b535709700335124"
format_stamp: "Formatted at 2026-09-28 23:17:05 on dist-test-slave-djkr"
server_key: "144a2101f8caa0fb05342ac3efd2a972"
server_key_iv: "a587e68a8e34bb73533299959b06ca84"
server_key_version: "encryptionkey@0"
I20260928 23:17:05.914405 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "8de16dd417064140b535709700335124"
format_stamp: "Formatted at 2026-09-28 23:17:05 on dist-test-slave-djkr"
server_key: "144a2101f8caa0fb05342ac3efd2a972"
server_key_iv: "a587e68a8e34bb73533299959b06ca84"
server_key_version: "encryptionkey@0"
I20260928 23:17:05.915663 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:05.916316 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:05.916464 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:05.916544 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "8de16dd417064140b535709700335124"
format_stamp: "Formatted at 2026-09-28 23:17:05 on dist-test-slave-djkr"
server_key: "144a2101f8caa0fb05342ac3efd2a972"
server_key_iv: "a587e68a8e34bb73533299959b06ca84"
server_key_version: "encryptionkey@0"
I20260928 23:17:05.916608 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:05.931523 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:05.933360 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:05.933488 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:05.937914 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34319
I20260928 23:17:05.937961 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34319 every 8 connection(s)
I20260928 23:17:05.938390 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:05.938499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 1472
I20260928 23:17:05.938578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:05.938745 392 external_mini_cluster.cc:1428] Setting key 3e600b2bd2e08ad12f1e00e9c5f88358
I20260928 23:17:05.939154 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:05.942206 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124: Bootstrap starting.
I20260928 23:17:05.942766 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:05.943023 1550 log.cc:824] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:05.943651 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124: No bootstrap required, opened a new log
Sep 28 23:17:05 dist-test-slave-djkr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:05.945926 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de16dd417064140b535709700335124" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260928 23:17:05.946136 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:05.946204 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de16dd417064140b535709700335124, State: Initialized, Role: FOLLOWER
I20260928 23:17:05.946331 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de16dd417064140b535709700335124" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260928 23:17:05.946429 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:05.946466 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:05.946517 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:05.948979 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:05.939280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35826 (local address 127.0.98.62:34319)
0928 23:17:05.939433 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:05.939442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:05.939470 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:17:05.939947 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:05.939956 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:05.940017 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:05.940161 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:05.940751 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:05.942122 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:05.942945 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:05.943388 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:05.943486 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:05.945966 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:05.945970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:05.945971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:05.945983 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:05.947796 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:05.948087 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:05.948089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:05.948090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:05.948161 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:05.948268 (+ 107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:05.948269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:05.948269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:05.948478 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:05.948574 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:05.948646 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:17:05.948745 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260928 23:17:05.979593 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de16dd417064140b535709700335124" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260928 23:17:05.979750 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8de16dd417064140b535709700335124; no voters:
I20260928 23:17:05.979952 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:05.980010 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:05.980167 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [term 1 LEADER]: Becoming Leader. State: Replica: 8de16dd417064140b535709700335124, State: Running, Role: LEADER
I20260928 23:17:05.980226 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:05.980283 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de16dd417064140b535709700335124" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } }
I20260928 23:17:05.980648 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8de16dd417064140b535709700335124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de16dd417064140b535709700335124" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } } }
I20260928 23:17:05.980902 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:05.980840 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8de16dd417064140b535709700335124. Latest consensus state: current_term: 1 leader_uuid: "8de16dd417064140b535709700335124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de16dd417064140b535709700335124" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34319 } } }
I20260928 23:17:05.981143 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124 [sys.catalog]: This master's current role is: LEADER
W20260928 23:17:05.981907 1571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:17:05.982004 1571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:17:05.982064 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:05.982255 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:05.984256 1564 catalog_manager.cc:1347] Generated new cluster ID: 515efde04244489bb50b80a3c2d84800
I20260928 23:17:05.984308 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:05.999061 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:05.999625 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:06.002974 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124: Generated new TSK 0
I20260928 23:17:06.003124 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260928 23:17:07.088912 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 1.082s user 0.000s sys 0.002s
I20260928 23:17:07.089206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:07.166422 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:07.166615 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:07.166636 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:07.167763 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:07.167804 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:07.167822 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:17:07.167838 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:07.169258 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:07.169445 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:07.169617 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:07.174288 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:07.174387 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:07.174497 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:07.174340 1576 server_base.cc:1034] running on GCE node
I20260928 23:17:07.174672 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:07.174881 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:07.176012 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790637427175994 us; error 30 us; skew 500 ppm
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:07.178262 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:17:07.178644 1576 webserver.cc:460] Webserver started at http://127.0.98.1:41799/ using document root <none> and password file <none>
I20260928 23:17:07.178838 1576 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:07.178884 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:07.179003 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:07.180078 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "59b1bd5113a54539bf7c8a761e6de7fa"
format_stamp: "Formatted at 2026-09-28 23:17:07 on dist-test-slave-djkr"
server_key: "f44101291aa266a2c20894806c8a5632"
server_key_iv: "e5e73cdeacb99f72b37fa0a732866c2b"
server_key_version: "encryptionkey@0"
I20260928 23:17:07.180418 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "59b1bd5113a54539bf7c8a761e6de7fa"
format_stamp: "Formatted at 2026-09-28 23:17:07 on dist-test-slave-djkr"
server_key: "f44101291aa266a2c20894806c8a5632"
server_key_iv: "e5e73cdeacb99f72b37fa0a732866c2b"
server_key_version: "encryptionkey@0"
I20260928 23:17:07.181646 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:07.182291 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.182446 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:07.182534 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "59b1bd5113a54539bf7c8a761e6de7fa"
format_stamp: "Formatted at 2026-09-28 23:17:07 on dist-test-slave-djkr"
server_key: "f44101291aa266a2c20894806c8a5632"
server_key_iv: "e5e73cdeacb99f72b37fa0a732866c2b"
server_key_version: "encryptionkey@0"
I20260928 23:17:07.182590 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:07.199100 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:07.201184 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:07.201326 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:07.201797 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:07.201843 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.201869 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:07.201885 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.208300 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38547
I20260928 23:17:07.208390 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38547 every 8 connection(s)
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:07.208703 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:07.211177 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.202233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47033 (local address 127.0.98.62:34319)
0928 23:17:07.202340 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.202342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.202505 (+ 163us) server_negotiation.cc:407] Connection header received
0928 23:17:07.203004 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.203006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.203019 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.203085 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:07.203976 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.204754 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.206227 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.206542 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.206600 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.208987 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:07.208989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:07.208989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:07.208998 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:07.210100 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.210496 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.210497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.210497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.210542 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.210707 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.210707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.210708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.210799 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:07.210851 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.211053 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.211111 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":55,"threads_started":1}
I20260928 23:17:07.211366 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.202229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:47033)
0928 23:17:07.202383 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.202425 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.202904 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.203200 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.203205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.203584 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:07.203878 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.203884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.204840 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.204841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.206138 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.206142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.206621 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.206622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.206764 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.207263 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:07.207273 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:07.208893 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:07.210182 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.210188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.210190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.210437 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.210582 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.210582 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.210583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.210667 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.210849 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:07.210853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:07.210957 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.211042 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.211155 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260928 23:17:07.211826 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260928 23:17:07.211898 1708 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:07.212059 1708 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260928 23:17:07.212500 1504 ts_manager.cc:194] Registered new tserver with Master: 59b1bd5113a54539bf7c8a761e6de7fa (127.0.98.1:38547)
I20260928 23:17:07.213148 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47033
I20260928 23:17:07.213416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 1576
I20260928 23:17:07.213496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:07.213665 392 external_mini_cluster.cc:1428] Setting key de6b2b0330884c88e822beaa46a07c18
I20260928 23:17:07.218055 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.212784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49551 (local address 127.0.98.62:34319)
0928 23:17:07.212817 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.212819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.212926 (+ 107us) server_negotiation.cc:407] Connection header received
0928 23:17:07.212980 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.212981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.212989 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.213028 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:07.213377 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.214027 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.214879 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.215062 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.215106 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.215915 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:07.215916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:07.215917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:07.215924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:07.217102 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.217435 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.217436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.217436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.217480 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.217653 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.217653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.217654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.217741 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:07.217784 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.217938 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.217996 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 23:17:07.218112 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.212734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.1:49551)
0928 23:17:07.212863 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.212873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.212929 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.213040 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.213041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.213188 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:07.213255 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.213257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.214116 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.214117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.214795 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.214798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.215117 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.215118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.215175 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.215533 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:07.215538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:07.215847 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:07.217178 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.217179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.217180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.217373 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.217533 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.217534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.217534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.217597 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.217790 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:07.217791 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:07.217841 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.217913 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.218009 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:17:07.236441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:07.315220 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:07.315430 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:07.315452 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:07.316588 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:07.316633 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:07.316653 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:17:07.316669 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:07.318104 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:07.318285 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:07.318461 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:07.323252 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:07.323289 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:07.323258 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:07.323436 1717 server_base.cc:1034] running on GCE node
I20260928 23:17:07.323676 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:07.323931 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:07.325094 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790637427325072 us; error 35 us; skew 500 ppm
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:07.327414 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:17:07.327855 1717 webserver.cc:460] Webserver started at http://127.0.98.2:46179/ using document root <none> and password file <none>
I20260928 23:17:07.328038 1717 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:07.328079 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:07.328178 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:07.329149 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da"
format_stamp: "Formatted at 2026-09-28 23:17:07 on dist-test-slave-djkr"
server_key: "ad22d818f0060c2038b05a27fe96a5d3"
server_key_iv: "5af0a8a195da5614c9565f0a42a06dd5"
server_key_version: "encryptionkey@0"
I20260928 23:17:07.329457 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da"
format_stamp: "Formatted at 2026-09-28 23:17:07 on dist-test-slave-djkr"
server_key: "ad22d818f0060c2038b05a27fe96a5d3"
server_key_iv: "5af0a8a195da5614c9565f0a42a06dd5"
server_key_version: "encryptionkey@0"
I20260928 23:17:07.330718 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:17:07.331301 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.331461 1717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:07.331538 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da"
format_stamp: "Formatted at 2026-09-28 23:17:07 on dist-test-slave-djkr"
server_key: "ad22d818f0060c2038b05a27fe96a5d3"
server_key_iv: "5af0a8a195da5614c9565f0a42a06dd5"
server_key_version: "encryptionkey@0"
I20260928 23:17:07.331627 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:07.349561 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:07.351423 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:07.351593 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:07.352087 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:07.352128 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.352155 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:07.352172 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:07.358565 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41273
I20260928 23:17:07.358631 1847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41273 every 8 connection(s)
I20260928 23:17:07.358911 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:07.360767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 1717
I20260928 23:17:07.360852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:07.361017 392 external_mini_cluster.cc:1428] Setting key 8708f232da2c260a129a700dd4bc8ff9
I20260928 23:17:07.361598 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.352526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57079 (local address 127.0.98.62:34319)
0928 23:17:07.352567 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.352569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.352781 (+ 212us) server_negotiation.cc:407] Connection header received
0928 23:17:07.353268 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.353269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.353280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.353338 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:07.354115 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.354866 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.356638 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.356924 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.357008 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.359018 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:07.359019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:07.359020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:07.359028 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:07.360325 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.360676 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.360678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.360678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.360733 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.360959 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.360959 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.360960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.361068 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:07.361125 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.361434 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.361507 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:17:07.361743 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.352520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:57079)
0928 23:17:07.352662 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.352696 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.353176 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.353417 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.353421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.353753 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:07.354031 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.354038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.354983 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.354984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.356528 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.356532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.357003 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.357004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.357107 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.357557 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:07.357566 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:07.358931 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:07.360404 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.360410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.360412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.360613 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.360784 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.360785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.360785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.360904 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.361121 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:07.361128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:07.361284 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.361398 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.361525 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260928 23:17:07.363217 1848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260928 23:17:07.363297 1848 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:07.363464 1848 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260928 23:17:07.363802 1504 ts_manager.cc:194] Registered new tserver with Master: 05fdee4bfde542ecbbdcf7f2ab0f96da (127.0.98.2:41273)
I20260928 23:17:07.364162 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57079
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:17:07.369841 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.364242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.2:43645)
0928 23:17:07.364370 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.364388 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.364469 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.364611 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.364611 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.364789 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:07.364853 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.364855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.365465 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.365466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.366261 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.366263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.366696 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.366697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.366742 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.367111 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:07.367124 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:07.367430 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:07.368907 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.368907 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.368908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.369069 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.369251 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.369252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.369252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.369296 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.369597 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:07.369598 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:07.369642 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.369691 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.369764 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 23:17:07.369884 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.364331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43645 (local address 127.0.98.62:34319)
0928 23:17:07.364374 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.364376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.364445 (+ 69us) server_negotiation.cc:407] Connection header received
0928 23:17:07.364531 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.364531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.364540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.364563 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:07.364923 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.365374 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.366336 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.366620 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.366684 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.367504 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:07.367506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:07.367506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:07.367512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:07.368823 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.369136 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.369137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.369137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.369190 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.369345 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.369346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.369346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.369536 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:07.369601 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.369724 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.369797 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:17:07.384361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34319
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:07.461508 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:07.461695 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:07.461717 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:07.462798 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:07.462831 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:07.462848 1857 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:17:07.462864 1857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:07.464351 1857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:07.464521 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:07.464704 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:07.469403 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:07.469425 1857 server_base.cc:1034] running on GCE node
W20260928 23:17:07.469419 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:07.469408 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:07.469859 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:07.470080 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:07.471221 1857 hybrid_clock.cc:648] HybridClock initialized: now 1790637427471197 us; error 38 us; skew 500 ppm
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:07.473573 1857 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:17:07.474006 1857 webserver.cc:460] Webserver started at http://127.0.98.3:43673/ using document root <none> and password file <none>
I20260928 23:17:07.474222 1857 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:07.474277 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:07.474404 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:07.475394 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "e5ca0f21fbba4734ba6de03258130dfb"
format_stamp: "Formatted at 2026-09-28 23:17:07 on dist-test-slave-djkr"
server_key: "bb29f847c2b46ca78e9b393b70991430"
server_key_iv: "797aa5fe1a9876b07f64f2595d359c23"
server_key_version: "encryptionkey@0"
I20260928 23:17:07.475749 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e5ca0f21fbba4734ba6de03258130dfb"
format_stamp: "Formatted at 2026-09-28 23:17:07 on dist-test-slave-djkr"
server_key: "bb29f847c2b46ca78e9b393b70991430"
server_key_iv: "797aa5fe1a9876b07f64f2595d359c23"
server_key_version: "encryptionkey@0"
I20260928 23:17:07.476914 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:07.477603 1872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.477748 1857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:07.477823 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "e5ca0f21fbba4734ba6de03258130dfb"
format_stamp: "Formatted at 2026-09-28 23:17:07 on dist-test-slave-djkr"
server_key: "bb29f847c2b46ca78e9b393b70991430"
server_key_iv: "797aa5fe1a9876b07f64f2595d359c23"
server_key_version: "encryptionkey@0"
I20260928 23:17:07.477882 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:07.491834 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:07.493525 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:07.493651 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:07.494144 1857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:07.494184 1857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.494211 1857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:07.494228 1857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:07.500820 1857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35853
I20260928 23:17:07.500867 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35853 every 8 connection(s)
I20260928 23:17:07.501211 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:07.503180 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.494507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51747 (local address 127.0.98.62:34319)
0928 23:17:07.494565 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.494567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.494800 (+ 233us) server_negotiation.cc:407] Connection header received
0928 23:17:07.495422 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.495423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.495433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.495480 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:07.496296 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.496824 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.497981 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.498285 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.498356 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.500775 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:07.500777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:07.500777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:07.500785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:07.502152 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.502527 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.502528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.502528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.502574 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.502729 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.502729 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.502729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.502818 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:07.502860 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.503062 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.503116 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:17:07.503394 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.494491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:51747)
0928 23:17:07.494650 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.494698 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.495322 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.495581 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.495586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.495930 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:07.496212 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.496218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.496907 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.496908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.497887 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.497890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.498393 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.498394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.498533 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.499062 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:07.499071 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:07.500683 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:07.502220 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.502224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.502225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.502470 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.502611 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.502611 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.502611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.502691 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.502860 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:07.502862 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:07.502967 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.503044 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.503155 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260928 23:17:07.503803 1988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34319
I20260928 23:17:07.503876 1988 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:07.504041 1988 heartbeater.cc:511] Master 127.0.98.62:34319 requested a full tablet report, sending...
I20260928 23:17:07.504390 1503 ts_manager.cc:194] Registered new tserver with Master: e5ca0f21fbba4734ba6de03258130dfb (127.0.98.3:35853)
I20260928 23:17:07.504863 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51747
I20260928 23:17:07.508481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 1857
I20260928 23:17:07.508570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:07.508745 392 external_mini_cluster.cc:1428] Setting key 9103d26de89e468da4b113115ab33e1a
I20260928 23:17:07.509505 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:17:07.509850 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.504699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.98.3:51493)
0928 23:17:07.504811 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.504821 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.504859 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.505071 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.505072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.505227 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:07.505285 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.505287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.506001 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.506002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.506516 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.506517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.506812 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.506813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.506873 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.507245 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:07.507250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:07.507537 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:07.508936 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.508937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.508938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.509132 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.509313 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.509314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.509314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.509377 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.509599 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:07.509599 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:07.509641 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.509696 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.509772 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260928 23:17:07.509862 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.504749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51493 (local address 127.0.98.62:34319)
0928 23:17:07.504982 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.504984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.504991 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:07.505007 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.505008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.505016 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.505065 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:07.505356 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.505923 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.506572 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.506735 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.506774 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.507643 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:07.507645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:07.507645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:07.507650 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:07.508864 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.509197 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.509198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.509199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.509255 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.509432 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.509433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.509433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.509547 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:07.509606 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.509727 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.509788 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:07.529923 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.523942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.0.98.62:34319)
0928 23:17:07.523986 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.523988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.524016 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:17:07.524071 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.524072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.524081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.524123 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:07.524622 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.525117 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.525815 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.525994 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.526048 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.527939 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:07.527941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:07.527941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:07.527947 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:07.529112 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.529388 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.529389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.529389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.529444 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.529577 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.529577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.529577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.529666 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:07.529733 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.529795 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.529851 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:17:07.531281 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:07.532852 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:07.538874 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.534996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38547 (local address 127.0.0.1:50824)
0928 23:17:07.535335 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.535375 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.535442 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.535741 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.535744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.535746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:07.535812 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.535815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.536644 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.536645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.537771 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.537774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.538583 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.538583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.538657 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.538671 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.538747 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.538803 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260928 23:17:07.538914 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.535115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.0.98.1:38547)
0928 23:17:07.535298 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.535301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.535528 (+ 227us) server_negotiation.cc:407] Connection header received
0928 23:17:07.535588 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.535590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.535605 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.535668 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:07.535889 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.536568 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.537854 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.538502 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.538573 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.538644 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.538766 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.538805 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260928 23:17:07.539280 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.535212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41273 (local address 127.0.0.1:47364)
0928 23:17:07.535485 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.535495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.535574 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.535816 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.535817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.535818 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:07.535844 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.535845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.537042 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.537043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.538016 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.538018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.539065 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.539066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.539134 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.539138 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.539182 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.539218 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":62,"threads_started":1}
I20260928 23:17:07.539525 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.535346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.0.98.2:41273)
0928 23:17:07.535490 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.535494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.535645 (+ 151us) server_negotiation.cc:407] Connection header received
0928 23:17:07.535681 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.535683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.535695 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.535757 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:07.535928 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.536959 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.538086 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.538785 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.538845 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.538914 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.539383 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.539434 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260928 23:17:07.539532 1642 tablet_service.cc:1467] Processing CreateTablet for tablet be09ab22195747b2811cfd52b4c88ea8 (DEFAULT_TABLE table=test-table [id=53275cd6613f48c4a5062758f7082f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:07.539978 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be09ab22195747b2811cfd52b4c88ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:07.540037 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.535083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.0.98.3:35853)
0928 23:17:07.535226 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.535231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.535436 (+ 205us) server_negotiation.cc:407] Connection header received
0928 23:17:07.535582 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.535584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.535602 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.535670 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:07.535892 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.536967 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.538883 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.539681 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.539745 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.539814 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.539907 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.539944 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260928 23:17:07.539963 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.534990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35853 (local address 127.0.0.1:47560)
0928 23:17:07.535335 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.535375 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.535460 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.535742 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.535745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.535747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:07.535818 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.535820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.538079 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.538079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.538804 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.538805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.539747 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.539748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.539806 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.539810 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.539864 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.539901 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260928 23:17:07.540655 1922 tablet_service.cc:1467] Processing CreateTablet for tablet be09ab22195747b2811cfd52b4c88ea8 (DEFAULT_TABLE table=test-table [id=53275cd6613f48c4a5062758f7082f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:07.540949 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be09ab22195747b2811cfd52b4c88ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:07.542881 2012 tablet_bootstrap.cc:492] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: Bootstrap starting.
I20260928 23:17:07.543358 1782 tablet_service.cc:1467] Processing CreateTablet for tablet be09ab22195747b2811cfd52b4c88ea8 (DEFAULT_TABLE table=test-table [id=53275cd6613f48c4a5062758f7082f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:07.543574 2012 tablet_bootstrap.cc:654] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:07.543706 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be09ab22195747b2811cfd52b4c88ea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:07.543725 2013 tablet_bootstrap.cc:492] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: Bootstrap starting.
I20260928 23:17:07.543951 2012 log.cc:824] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:07.544306 2013 tablet_bootstrap.cc:654] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:07.544571 2012 tablet_bootstrap.cc:492] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: No bootstrap required, opened a new log
I20260928 23:17:07.544596 2013 log.cc:824] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:07.544627 2012 ts_tablet_manager.cc:1397] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:07.545306 2013 tablet_bootstrap.cc:492] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: No bootstrap required, opened a new log
I20260928 23:17:07.545372 2013 ts_tablet_manager.cc:1397] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:07.546120 2012 raft_consensus.cc:348] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } }
I20260928 23:17:07.546280 2012 raft_consensus.cc:374] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:07.546329 2012 raft_consensus.cc:729] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b1bd5113a54539bf7c8a761e6de7fa, State: Initialized, Role: FOLLOWER
I20260928 23:17:07.546453 2012 consensus_queue.cc:260] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } }
I20260928 23:17:07.546680 2012 ts_tablet_manager.cc:1428] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:17:07.546797 1708 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
I20260928 23:17:07.547372 2017 tablet_bootstrap.cc:492] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: Bootstrap starting.
I20260928 23:17:07.547214 2013 raft_consensus.cc:348] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } }
I20260928 23:17:07.547452 2013 raft_consensus.cc:374] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:07.547497 2013 raft_consensus.cc:729] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ca0f21fbba4734ba6de03258130dfb, State: Initialized, Role: FOLLOWER
I20260928 23:17:07.547624 2013 consensus_queue.cc:260] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } }
I20260928 23:17:07.547879 2013 ts_tablet_manager.cc:1428] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:17:07.547935 2017 tablet_bootstrap.cc:654] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:07.548018 1988 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
I20260928 23:17:07.548211 2017 log.cc:824] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:07.548861 2017 tablet_bootstrap.cc:492] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: No bootstrap required, opened a new log
I20260928 23:17:07.548916 2017 ts_tablet_manager.cc:1397] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:07.550119 2017 raft_consensus.cc:348] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } }
I20260928 23:17:07.550230 2017 raft_consensus.cc:374] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:07.550248 2017 raft_consensus.cc:729] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fdee4bfde542ecbbdcf7f2ab0f96da, State: Initialized, Role: FOLLOWER
I20260928 23:17:07.550331 2017 consensus_queue.cc:260] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } }
I20260928 23:17:07.550570 2017 ts_tablet_manager.cc:1428] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:17:07.550599 1848 heartbeater.cc:503] Master 127.0.98.62:34319 was elected leader, sending a full tablet report...
W20260928 23:17:07.609678 1849 tablet.cc:2369] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:07.643867 2020 raft_consensus.cc:482] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:07.643986 2020 raft_consensus.cc:504] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } }
I20260928 23:17:07.644279 2020 leader_election.cc:288] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5ca0f21fbba4734ba6de03258130dfb (127.0.98.3:35853), 59b1bd5113a54539bf7c8a761e6de7fa (127.0.98.1:38547)
I20260928 23:17:07.646763 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.644474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35853 (local address 127.0.98.2:48471)
0928 23:17:07.644522 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.644534 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.644580 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.644685 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.644686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.644687 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:07.644750 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.644752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.645355 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.645356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.646070 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.646071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.646545 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.646546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.646604 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.646606 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.646649 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.646685 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:17:07.646754 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.644547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48471 (local address 127.0.98.3:35853)
0928 23:17:07.644587 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.644589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.644599 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:07.644626 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.644627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.644636 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.644683 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:07.644803 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.645285 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.646145 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.646480 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.646537 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.646547 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.646658 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.646689 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:17:07.646981 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.644599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57349 (local address 127.0.98.1:38547)
0928 23:17:07.644640 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.644642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.644706 (+ 64us) server_negotiation.cc:407] Connection header received
0928 23:17:07.644768 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.644769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.644780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.644828 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:07.644958 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.645437 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.646314 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.646651 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.646706 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.646716 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.646874 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.646918 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:17:07.647063 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be09ab22195747b2811cfd52b4c88ea8" candidate_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ca0f21fbba4734ba6de03258130dfb" is_pre_election: true
I20260928 23:17:07.647197 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.644488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38547 (local address 127.0.98.2:57349)
0928 23:17:07.644641 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.644650 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.644719 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.644836 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.644837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.644838 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:07.644907 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.644908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.645513 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.645513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.646237 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.646239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.646724 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.646725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.646788 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.646791 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.647064 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.647114 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260928 23:17:07.647225 1942 raft_consensus.cc:2454] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fdee4bfde542ecbbdcf7f2ab0f96da in term 0.
I20260928 23:17:07.647478 1733 leader_election.cc:302] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fdee4bfde542ecbbdcf7f2ab0f96da, e5ca0f21fbba4734ba6de03258130dfb; no voters:
I20260928 23:17:07.647498 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be09ab22195747b2811cfd52b4c88ea8" candidate_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" is_pre_election: true
I20260928 23:17:07.647625 2020 raft_consensus.cc:2781] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:07.647641 1662 raft_consensus.cc:2454] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fdee4bfde542ecbbdcf7f2ab0f96da in term 0.
I20260928 23:17:07.647670 2020 raft_consensus.cc:482] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:07.647701 2020 raft_consensus.cc:3037] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:07.648705 2020 raft_consensus.cc:504] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } }
I20260928 23:17:07.648831 2020 leader_election.cc:288] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [CANDIDATE]: Term 1 election: Requested vote from peers e5ca0f21fbba4734ba6de03258130dfb (127.0.98.3:35853), 59b1bd5113a54539bf7c8a761e6de7fa (127.0.98.1:38547)
I20260928 23:17:07.648967 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be09ab22195747b2811cfd52b4c88ea8" candidate_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b1bd5113a54539bf7c8a761e6de7fa"
I20260928 23:17:07.648978 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be09ab22195747b2811cfd52b4c88ea8" candidate_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ca0f21fbba4734ba6de03258130dfb"
I20260928 23:17:07.649037 1942 raft_consensus.cc:3037] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:07.649040 1662 raft_consensus.cc:3037] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:07.649681 1942 raft_consensus.cc:2454] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fdee4bfde542ecbbdcf7f2ab0f96da in term 1.
I20260928 23:17:07.649712 1662 raft_consensus.cc:2454] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fdee4bfde542ecbbdcf7f2ab0f96da in term 1.
I20260928 23:17:07.649845 1736 leader_election.cc:302] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fdee4bfde542ecbbdcf7f2ab0f96da, 59b1bd5113a54539bf7c8a761e6de7fa; no voters:
I20260928 23:17:07.649921 2020 raft_consensus.cc:2781] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:07.650038 2020 raft_consensus.cc:686] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 LEADER]: Becoming Leader. State: Replica: 05fdee4bfde542ecbbdcf7f2ab0f96da, State: Running, Role: LEADER
I20260928 23:17:07.650187 2020 consensus_queue.cc:237] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } }
I20260928 23:17:07.650918 1503 catalog_manager.cc:5495] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da reported cstate change: term changed from 0 to 1, leader changed from <none> to 05fdee4bfde542ecbbdcf7f2ab0f96da (127.0.98.2). New cstate: current_term: 1 leader_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } health_report { overall_health: UNKNOWN } } }
I20260928 23:17:07.682433 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.680060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.0.98.2:41273)
0928 23:17:07.680112 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.680114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.680122 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:07.680153 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.680154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.680171 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.680225 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:07.680357 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.680845 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.681725 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.681936 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.681985 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.682135 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:07.682239 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:07.682289 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.682350 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.682367 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:17:07.684067 1942 raft_consensus.cc:1261] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb [term 1 FOLLOWER]: Refusing update from remote peer 05fdee4bfde542ecbbdcf7f2ab0f96da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:07.684329 1662 raft_consensus.cc:1261] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa [term 1 FOLLOWER]: Refusing update from remote peer 05fdee4bfde542ecbbdcf7f2ab0f96da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:07.684415 2022 consensus_queue.cc:1035] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:07.684541 2020 consensus_queue.cc:1035] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:07.686578 2025 mvcc.cc:205] Tried to move back new op lower bound from 7334450903791325184 to 7334450903655632896. Current Snapshot: MvccSnapshot[applied={T|T < 7334450903791325184}]
I20260928 23:17:07.686820 2026 mvcc.cc:205] Tried to move back new op lower bound from 7334450903791325184 to 7334450903655632896. Current Snapshot: MvccSnapshot[applied={T|T < 7334450903791325184}]
I20260928 23:17:07.687390 2027 mvcc.cc:205] Tried to move back new op lower bound from 7334450903791325184 to 7334450903655632896. Current Snapshot: MvccSnapshot[applied={T|T < 7334450903791325184}]
I20260928 23:17:07.689697 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:07.689774 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:07.690436 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124: Sending DeleteTablet for 3 replicas of tablet be09ab22195747b2811cfd52b4c88ea8
I20260928 23:17:07.690721 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet be09ab22195747b2811cfd52b4c88ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47560
I20260928 23:17:07.690755 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet be09ab22195747b2811cfd52b4c88ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50824
I20260928 23:17:07.690752 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet be09ab22195747b2811cfd52b4c88ea8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47364
I20260928 23:17:07.690945 2035 tablet_replica.cc:330] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: stopping tablet replica
I20260928 23:17:07.691033 2035 raft_consensus.cc:2229] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:07.691147 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:07.691287 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:07.691366 2035 raft_consensus.cc:2258] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:07.691577 2036 tablet_replica.cc:330] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: stopping tablet replica
I20260928 23:17:07.691821 2035 ts_tablet_manager.cc:1905] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:07.692250 2034 tablet_replica.cc:330] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: stopping tablet replica
I20260928 23:17:07.692478 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 646ba3c06bab4a0d87150deabcf7317e (DEFAULT_TABLE table=test-table [id=5ee99d85a09e45f88d966151a2df6efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:07.692541 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 646ba3c06bab4a0d87150deabcf7317e (DEFAULT_TABLE table=test-table [id=5ee99d85a09e45f88d966151a2df6efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:07.692570 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646ba3c06bab4a0d87150deabcf7317e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:07.692628 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646ba3c06bab4a0d87150deabcf7317e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:07.693719 2013 tablet_bootstrap.cc:492] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: Bootstrap starting.
I20260928 23:17:07.692443 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 646ba3c06bab4a0d87150deabcf7317e (DEFAULT_TABLE table=test-table [id=5ee99d85a09e45f88d966151a2df6efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:07.694133 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 646ba3c06bab4a0d87150deabcf7317e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:07.694141 2013 tablet_bootstrap.cc:654] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:07.694449 2012 tablet_bootstrap.cc:492] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: Bootstrap starting.
I20260928 23:17:07.695029 2012 tablet_bootstrap.cc:654] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:07.695041 2035 ts_tablet_manager.cc:1918] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:07.695099 2035 log.cc:1196] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/be09ab22195747b2811cfd52b4c88ea8
I20260928 23:17:07.695287 2017 tablet_bootstrap.cc:492] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da: Bootstrap starting.
I20260928 23:17:07.695386 2036 raft_consensus.cc:2229] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:07.695380 2035 ts_tablet_manager.cc:1939] T be09ab22195747b2811cfd52b4c88ea8 P e5ca0f21fbba4734ba6de03258130dfb: Deleting consensus metadata
I20260928 23:17:07.695483 2036 raft_consensus.cc:2258] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:07.695703 2017 tablet_bootstrap.cc:654] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:07.695749 1488 catalog_manager.cc:4841] TS e5ca0f21fbba4734ba6de03258130dfb (127.0.98.3:35853): tablet be09ab22195747b2811cfd52b4c88ea8 (table test-table [id=53275cd6613f48c4a5062758f7082f2a]) successfully deleted
I20260928 23:17:07.695953 2034 raft_consensus.cc:2229] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:07.696064 2034 raft_consensus.cc:2258] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:07.696130 2012 tablet_bootstrap.cc:492] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: No bootstrap required, opened a new log
I20260928 23:17:07.696173 2012 ts_tablet_manager.cc:1397] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:07.696324 2012 raft_consensus.cc:348] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.696398 2012 raft_consensus.cc:374] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:07.696429 2012 raft_consensus.cc:729] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b1bd5113a54539bf7c8a761e6de7fa, State: Initialized, Role: FOLLOWER
I20260928 23:17:07.696450 2017 tablet_bootstrap.cc:492] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da: No bootstrap required, opened a new log
I20260928 23:17:07.696470 2012 consensus_queue.cc:260] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.696488 2017 ts_tablet_manager.cc:1397] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:07.696522 2034 ts_tablet_manager.cc:1905] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:07.696557 2012 ts_tablet_manager.cc:1428] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.696620 2017 raft_consensus.cc:348] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.696686 2017 raft_consensus.cc:374] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:07.696718 2017 raft_consensus.cc:729] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fdee4bfde542ecbbdcf7f2ab0f96da, State: Initialized, Role: FOLLOWER
I20260928 23:17:07.696756 2017 consensus_queue.cc:260] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.696834 2017 ts_tablet_manager.cc:1428] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.697814 2034 ts_tablet_manager.cc:1918] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:07.697867 2034 log.cc:1196] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/be09ab22195747b2811cfd52b4c88ea8
I20260928 23:17:07.698109 2034 ts_tablet_manager.cc:1939] T be09ab22195747b2811cfd52b4c88ea8 P 59b1bd5113a54539bf7c8a761e6de7fa: Deleting consensus metadata
I20260928 23:17:07.698462 2036 ts_tablet_manager.cc:1905] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:07.698556 1491 catalog_manager.cc:4841] TS 59b1bd5113a54539bf7c8a761e6de7fa (127.0.98.1:38547): tablet be09ab22195747b2811cfd52b4c88ea8 (table test-table [id=53275cd6613f48c4a5062758f7082f2a]) successfully deleted
I20260928 23:17:07.699618 2036 ts_tablet_manager.cc:1918] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:07.699683 2036 log.cc:1196] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/be09ab22195747b2811cfd52b4c88ea8
I20260928 23:17:07.699883 2036 ts_tablet_manager.cc:1939] T be09ab22195747b2811cfd52b4c88ea8 P 05fdee4bfde542ecbbdcf7f2ab0f96da: Deleting consensus metadata
I20260928 23:17:07.699899 2013 tablet_bootstrap.cc:492] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: No bootstrap required, opened a new log
I20260928 23:17:07.699940 2013 ts_tablet_manager.cc:1397] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 23:17:07.700075 2013 raft_consensus.cc:348] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.700155 2013 raft_consensus.cc:374] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:07.700174 2013 raft_consensus.cc:729] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ca0f21fbba4734ba6de03258130dfb, State: Initialized, Role: FOLLOWER
I20260928 23:17:07.700206 1488 catalog_manager.cc:4841] TS 05fdee4bfde542ecbbdcf7f2ab0f96da (127.0.98.2:41273): tablet be09ab22195747b2811cfd52b4c88ea8 (table test-table [id=53275cd6613f48c4a5062758f7082f2a]) successfully deleted
I20260928 23:17:07.700225 2013 consensus_queue.cc:260] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.700304 2013 ts_tablet_manager.cc:1428] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.705607 2018 raft_consensus.cc:482] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:07.705660 2018 raft_consensus.cc:504] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.705897 2018 leader_election.cc:288] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59b1bd5113a54539bf7c8a761e6de7fa (127.0.98.1:38547), 05fdee4bfde542ecbbdcf7f2ab0f96da (127.0.98.2:41273)
I20260928 23:17:07.709046 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.706117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57281 (local address 127.0.98.2:41273)
0928 23:17:07.706154 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.706156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.706285 (+ 129us) server_negotiation.cc:407] Connection header received
0928 23:17:07.706355 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.706356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.706364 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.706407 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:07.706597 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.707239 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.708189 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.708716 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.708783 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.708792 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.708959 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.708990 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:17:07.709084 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.706040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41273 (local address 127.0.98.3:57281)
0928 23:17:07.706231 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.706240 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.706304 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.706445 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.706446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.706448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:07.706529 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.706530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.707301 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.707302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.708116 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.708119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.708812 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.708813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.708887 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.708892 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.708949 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.708986 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":30,"thread_start_us":42,"threads_started":1}
I20260928 23:17:07.709235 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.706033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38547 (local address 127.0.98.3:45095)
0928 23:17:07.706069 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.706081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.706115 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.706284 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.706285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.706286 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:07.706348 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.706350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.707163 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.707163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.708377 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.708379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.709018 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.709019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.709094 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.709098 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.709142 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.709178 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 23:17:07.709517 1709 tablet.cc:2369] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:07.709494 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646ba3c06bab4a0d87150deabcf7317e" candidate_uuid: "e5ca0f21fbba4734ba6de03258130dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" is_pre_election: true
I20260928 23:17:07.709589 1802 raft_consensus.cc:2454] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5ca0f21fbba4734ba6de03258130dfb in term 0.
I20260928 23:17:07.709740 1873 leader_election.cc:302] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fdee4bfde542ecbbdcf7f2ab0f96da, e5ca0f21fbba4734ba6de03258130dfb; no voters:
I20260928 23:17:07.709794 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.706089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45095 (local address 127.0.98.1:38547)
0928 23:17:07.706181 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.706182 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.706200 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:17:07.706222 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.706222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.706230 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.706278 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:07.706402 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.707092 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.708455 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.708927 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.708983 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.708990 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.709726 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.709756 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 23:17:07.709856 2018 raft_consensus.cc:2781] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:07.709888 2018 raft_consensus.cc:482] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:07.709908 2018 raft_consensus.cc:3037] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:07.709965 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646ba3c06bab4a0d87150deabcf7317e" candidate_uuid: "e5ca0f21fbba4734ba6de03258130dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" is_pre_election: true
I20260928 23:17:07.710033 1662 raft_consensus.cc:2454] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5ca0f21fbba4734ba6de03258130dfb in term 0.
I20260928 23:17:07.710631 2018 raft_consensus.cc:504] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.710757 2018 leader_election.cc:288] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [CANDIDATE]: Term 1 election: Requested vote from peers 59b1bd5113a54539bf7c8a761e6de7fa (127.0.98.1:38547), 05fdee4bfde542ecbbdcf7f2ab0f96da (127.0.98.2:41273)
I20260928 23:17:07.710842 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646ba3c06bab4a0d87150deabcf7317e" candidate_uuid: "e5ca0f21fbba4734ba6de03258130dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b1bd5113a54539bf7c8a761e6de7fa"
I20260928 23:17:07.710912 1662 raft_consensus.cc:3037] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:07.710928 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "646ba3c06bab4a0d87150deabcf7317e" candidate_uuid: "e5ca0f21fbba4734ba6de03258130dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da"
I20260928 23:17:07.710995 1802 raft_consensus.cc:3037] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:07.711480 1662 raft_consensus.cc:2454] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5ca0f21fbba4734ba6de03258130dfb in term 1.
I20260928 23:17:07.711654 1876 leader_election.cc:302] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b1bd5113a54539bf7c8a761e6de7fa, e5ca0f21fbba4734ba6de03258130dfb; no voters:
I20260928 23:17:07.711723 2018 raft_consensus.cc:2781] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:07.711817 1802 raft_consensus.cc:2454] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5ca0f21fbba4734ba6de03258130dfb in term 1.
I20260928 23:17:07.711833 2018 raft_consensus.cc:686] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 1 LEADER]: Becoming Leader. State: Replica: e5ca0f21fbba4734ba6de03258130dfb, State: Running, Role: LEADER
I20260928 23:17:07.711889 2018 consensus_queue.cc:237] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.712443 1504 catalog_manager.cc:5495] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to e5ca0f21fbba4734ba6de03258130dfb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e5ca0f21fbba4734ba6de03258130dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } health_report { overall_health: HEALTHY } } }
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:07.734992 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.728230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34319 (local address 127.0.0.1:35836)
0928 23:17:07.728393 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.728418 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.728469 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.728583 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.728584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.728851 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:07.728892 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.728893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.729790 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.729790 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.730294 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.730295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.730719 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.730720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.730753 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.731201 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:07.731205 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:07.732496 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:07.734090 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.734096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.734099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.734278 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.734457 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:07.734457 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:07.734458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:07.734503 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:07.734737 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:07.734744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:07.734787 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.734837 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.734922 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20260928 23:17:07.735090 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.728282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.0.98.62:34319)
0928 23:17:07.728351 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.728353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.728474 (+ 121us) server_negotiation.cc:407] Connection header received
0928 23:17:07.728503 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.728504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.728513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.728581 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:07.728954 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.729725 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.730359 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.730651 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.730716 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.732569 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:07.732571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:07.732571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:07.732578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:07.733998 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.734340 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.734341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.734342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.734401 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.734548 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.734548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.734549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.734672 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:07.734736 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.734872 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.734982 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":45}
I20260928 23:17:07.737156 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35836:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:17:07.737380 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:07.738595 1782 tablet_service.cc:1467] Processing CreateTablet for tablet d1098eee1fbd4117b035c6f1cef2f03c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a21d62f9ad504982bddb80318fd81ab8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:17:07.738636 1642 tablet_service.cc:1467] Processing CreateTablet for tablet d1098eee1fbd4117b035c6f1cef2f03c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a21d62f9ad504982bddb80318fd81ab8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:17:07.738749 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1098eee1fbd4117b035c6f1cef2f03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:07.738761 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1098eee1fbd4117b035c6f1cef2f03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:07.740053 1922 tablet_service.cc:1467] Processing CreateTablet for tablet d1098eee1fbd4117b035c6f1cef2f03c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a21d62f9ad504982bddb80318fd81ab8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:17:07.740139 2017 tablet_bootstrap.cc:492] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: Bootstrap starting.
I20260928 23:17:07.740181 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1098eee1fbd4117b035c6f1cef2f03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:07.740516 2017 tablet_bootstrap.cc:654] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:07.741171 2017 tablet_bootstrap.cc:492] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: No bootstrap required, opened a new log
I20260928 23:17:07.741217 2017 ts_tablet_manager.cc:1397] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:07.741340 2017 raft_consensus.cc:348] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.741400 2017 raft_consensus.cc:374] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:07.741429 2017 raft_consensus.cc:729] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fdee4bfde542ecbbdcf7f2ab0f96da, State: Initialized, Role: FOLLOWER
I20260928 23:17:07.741470 2017 consensus_queue.cc:260] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.741505 2013 tablet_bootstrap.cc:492] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: Bootstrap starting.
I20260928 23:17:07.741516 2012 tablet_bootstrap.cc:492] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: Bootstrap starting.
I20260928 23:17:07.741564 2017 ts_tablet_manager.cc:1428] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.741638 2022 tablet_replica.cc:439] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } }
I20260928 23:17:07.741703 2022 tablet_replica.cc:442] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:07.741854 2012 tablet_bootstrap.cc:654] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:07.741890 2013 tablet_bootstrap.cc:654] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:07.742583 2012 tablet_bootstrap.cc:492] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: No bootstrap required, opened a new log
I20260928 23:17:07.742635 2012 ts_tablet_manager.cc:1397] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:07.742743 2013 tablet_bootstrap.cc:492] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: No bootstrap required, opened a new log
I20260928 23:17:07.742787 2013 ts_tablet_manager.cc:1397] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:07.742760 2012 raft_consensus.cc:348] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.742820 2012 raft_consensus.cc:374] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:07.742843 2012 raft_consensus.cc:729] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b1bd5113a54539bf7c8a761e6de7fa, State: Initialized, Role: FOLLOWER
I20260928 23:17:07.742887 2012 consensus_queue.cc:260] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.742970 2012 ts_tablet_manager.cc:1428] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:07.743049 2013 raft_consensus.cc:348] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.743109 2013 raft_consensus.cc:374] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:07.743129 2013 raft_consensus.cc:729] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ca0f21fbba4734ba6de03258130dfb, State: Initialized, Role: FOLLOWER
I20260928 23:17:07.743101 2016 tablet_replica.cc:439] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } }
I20260928 23:17:07.743158 2016 tablet_replica.cc:442] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:07.743170 2013 consensus_queue.cc:260] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.743243 2013 ts_tablet_manager.cc:1428] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:07.743278 2018 tablet_replica.cc:439] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } }
I20260928 23:17:07.743336 2018 tablet_replica.cc:442] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:07.748025 2022 raft_consensus.cc:482] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:07.748090 2022 raft_consensus.cc:504] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.748221 2022 leader_election.cc:288] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59b1bd5113a54539bf7c8a761e6de7fa (127.0.98.1:38547), e5ca0f21fbba4734ba6de03258130dfb (127.0.98.3:35853)
I20260928 23:17:07.748387 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1098eee1fbd4117b035c6f1cef2f03c" candidate_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ca0f21fbba4734ba6de03258130dfb" is_pre_election: true
I20260928 23:17:07.748420 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1098eee1fbd4117b035c6f1cef2f03c" candidate_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" is_pre_election: true
I20260928 23:17:07.748466 1942 raft_consensus.cc:2454] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fdee4bfde542ecbbdcf7f2ab0f96da in term 0.
I20260928 23:17:07.748485 1662 raft_consensus.cc:2454] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fdee4bfde542ecbbdcf7f2ab0f96da in term 0.
I20260928 23:17:07.748598 1733 leader_election.cc:302] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fdee4bfde542ecbbdcf7f2ab0f96da, e5ca0f21fbba4734ba6de03258130dfb; no voters:
I20260928 23:17:07.748716 2022 raft_consensus.cc:2781] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:07.748761 2022 raft_consensus.cc:482] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:07.748787 2022 raft_consensus.cc:3037] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:07.749413 2022 raft_consensus.cc:504] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.749533 2022 leader_election.cc:288] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [CANDIDATE]: Term 1 election: Requested vote from peers 59b1bd5113a54539bf7c8a761e6de7fa (127.0.98.1:38547), e5ca0f21fbba4734ba6de03258130dfb (127.0.98.3:35853)
I20260928 23:17:07.749657 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1098eee1fbd4117b035c6f1cef2f03c" candidate_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5ca0f21fbba4734ba6de03258130dfb"
I20260928 23:17:07.749665 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1098eee1fbd4117b035c6f1cef2f03c" candidate_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b1bd5113a54539bf7c8a761e6de7fa"
I20260928 23:17:07.749712 1942 raft_consensus.cc:3037] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:07.749720 1662 raft_consensus.cc:3037] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:07.750236 1942 raft_consensus.cc:2454] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fdee4bfde542ecbbdcf7f2ab0f96da in term 1.
I20260928 23:17:07.750363 1662 raft_consensus.cc:2454] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fdee4bfde542ecbbdcf7f2ab0f96da in term 1.
I20260928 23:17:07.750403 1733 leader_election.cc:302] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fdee4bfde542ecbbdcf7f2ab0f96da, e5ca0f21fbba4734ba6de03258130dfb; no voters:
I20260928 23:17:07.750547 2022 raft_consensus.cc:2781] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:07.750605 2022 raft_consensus.cc:686] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 LEADER]: Becoming Leader. State: Replica: 05fdee4bfde542ecbbdcf7f2ab0f96da, State: Running, Role: LEADER
I20260928 23:17:07.750638 2022 consensus_queue.cc:237] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } }
I20260928 23:17:07.750872 2020 tablet_replica.cc:439] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: TxnStatusTablet state changed. Reason: New leader 05fdee4bfde542ecbbdcf7f2ab0f96da. Latest consensus state: current_term: 1 leader_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } }
I20260928 23:17:07.750944 2020 tablet_replica.cc:442] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:17:07.751088 2053 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:17:07.751184 2053 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:17:07.751302 1503 catalog_manager.cc:5495] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da reported cstate change: term changed from 0 to 1, leader changed from <none> to 05fdee4bfde542ecbbdcf7f2ab0f96da (127.0.98.2). New cstate: current_term: 1 leader_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } health_report { overall_health: UNKNOWN } } }
W20260928 23:17:07.751714 1989 tablet.cc:2369] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:07.759687 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.757551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.0.98.2:41273)
0928 23:17:07.757591 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.757593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.757652 (+ 59us) server_negotiation.cc:407] Connection header received
0928 23:17:07.757725 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.757726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.757735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.757781 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:07.757897 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.758350 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.759060 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.759259 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.759311 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.759459 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:07.759491 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:07.759533 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.759601 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.759619 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:17:07.759698 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.757498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41273 (local address 127.0.0.1:47392)
0928 23:17:07.757608 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.757618 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.757671 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.757786 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.757787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.757788 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:17:07.757833 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.757836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.758431 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.758432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.758993 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.758994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.759320 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.759321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.759382 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.759396 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:17:07.759533 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:17:07.759538 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.759589 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.759632 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 23:17:07.760600 1662 raft_consensus.cc:1261] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa [term 1 FOLLOWER]: Refusing update from remote peer 05fdee4bfde542ecbbdcf7f2ab0f96da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:07.760633 1942 raft_consensus.cc:1261] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb [term 1 FOLLOWER]: Refusing update from remote peer 05fdee4bfde542ecbbdcf7f2ab0f96da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:07.760795 2020 consensus_queue.cc:1035] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:07.760886 2022 consensus_queue.cc:1035] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:07.761348 2016 tablet_replica.cc:439] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: TxnStatusTablet state changed. Reason: New leader 05fdee4bfde542ecbbdcf7f2ab0f96da. Latest consensus state: current_term: 1 leader_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } }
I20260928 23:17:07.761425 2016 tablet_replica.cc:442] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:07.761462 2018 tablet_replica.cc:439] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: TxnStatusTablet state changed. Reason: New leader 05fdee4bfde542ecbbdcf7f2ab0f96da. Latest consensus state: current_term: 1 leader_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } }
I20260928 23:17:07.761554 2018 tablet_replica.cc:442] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:07.761727 2020 tablet_replica.cc:439] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } }
I20260928 23:17:07.761797 2020 tablet_replica.cc:442] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:17:07.762109 2018 tablet_replica.cc:439] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } }
I20260928 23:17:07.762183 2018 tablet_replica.cc:442] T d1098eee1fbd4117b035c6f1cef2f03c P e5ca0f21fbba4734ba6de03258130dfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:07.762750 2022 tablet_replica.cc:439] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } }
I20260928 23:17:07.762827 2022 tablet_replica.cc:442] T d1098eee1fbd4117b035c6f1cef2f03c P 05fdee4bfde542ecbbdcf7f2ab0f96da: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:17:07.763094 2016 tablet_replica.cc:439] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 } } peers { permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 } } peers { permanent_uuid: "e5ca0f21fbba4734ba6de03258130dfb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35853 } } }
I20260928 23:17:07.763164 2016 tablet_replica.cc:442] T d1098eee1fbd4117b035c6f1cef2f03c P 59b1bd5113a54539bf7c8a761e6de7fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:07.765909 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.763811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.0.98.3:35853)
0928 23:17:07.763842 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.763844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.763851 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:07.763931 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.763932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.763948 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.764003 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:07.764098 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.764566 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.765252 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.765448 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.765499 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.765632 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:07.765724 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:07.765782 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.765830 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.765850 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:17:07.770006 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.768081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33067 (local address 127.0.98.2:41273)
0928 23:17:07.768122 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.768123 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.768132 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:07.768151 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.768152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.768160 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.768209 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:07.768302 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.768735 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.769416 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.769604 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.769654 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.769795 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:07.769826 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:07.769868 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.769929 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.769946 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:17:07.770021 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.768012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41273 (local address 127.0.98.3:33067)
0928 23:17:07.768072 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.768082 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.768114 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.768212 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.768212 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.768213 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:17:07.768261 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.768262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.768799 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.768800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.769348 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.769349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.769657 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.769658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.769721 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.769733 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:17:07.769887 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:17:07.769889 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.769927 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.769962 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 23:17:07.772073 1662 raft_consensus.cc:1261] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa [term 1 FOLLOWER]: Refusing update from remote peer e5ca0f21fbba4734ba6de03258130dfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:07.772100 1802 raft_consensus.cc:1261] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 FOLLOWER]: Refusing update from remote peer e5ca0f21fbba4734ba6de03258130dfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:07.772286 2041 consensus_queue.cc:1035] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b1bd5113a54539bf7c8a761e6de7fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:07.772393 2018 consensus_queue.cc:1035] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "05fdee4bfde542ecbbdcf7f2ab0f96da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:07.779402 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.777219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36097 (local address 127.0.98.3:35853)
0928 23:17:07.777252 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.777253 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.777260 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:07.777277 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.777278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.777287 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.777331 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:07.777530 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.777960 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.778637 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.778917 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.778969 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.779174 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:07.779203 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:07.779244 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.779316 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.779337 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:17:07.779428 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.777018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35853 (local address 127.0.98.2:36097)
0928 23:17:07.777113 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:07.777123 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:07.777159 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:07.777411 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:07.777412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:07.777413 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:17:07.777471 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.777472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.778022 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.778023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.778571 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:07.778572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.779017 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.779018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:07.779078 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.779094 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:17:07.779266 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:17:07.779268 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:17:07.779307 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 23:17:07.779364 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 23:17:07.786073 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:07.786160 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:07.786700 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8de16dd417064140b535709700335124: Sending DeleteTablet for 3 replicas of tablet 646ba3c06bab4a0d87150deabcf7317e
I20260928 23:17:07.786916 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 646ba3c06bab4a0d87150deabcf7317e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50824
I20260928 23:17:07.787021 2034 tablet_replica.cc:330] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: stopping tablet replica
I20260928 23:17:07.787073 2034 raft_consensus.cc:2229] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:07.787110 2034 raft_consensus.cc:2258] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:07.787453 2034 ts_tablet_manager.cc:1905] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:07.788636 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 646ba3c06bab4a0d87150deabcf7317e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47364
I20260928 23:17:07.788735 2036 tablet_replica.cc:330] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da: stopping tablet replica
I20260928 23:17:07.788774 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet 646ba3c06bab4a0d87150deabcf7317e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47560
I20260928 23:17:07.788787 2036 raft_consensus.cc:2229] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:07.788817 2036 raft_consensus.cc:2258] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:07.788836 2035 tablet_replica.cc:330] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: stopping tablet replica
I20260928 23:17:07.788887 2035 raft_consensus.cc:2229] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:07.788967 2035 raft_consensus.cc:2258] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:07.789163 2034 ts_tablet_manager.cc:1918] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:07.789215 2034 log.cc:1196] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/646ba3c06bab4a0d87150deabcf7317e
I20260928 23:17:07.789271 2035 ts_tablet_manager.cc:1905] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:07.789397 2034 ts_tablet_manager.cc:1939] T 646ba3c06bab4a0d87150deabcf7317e P 59b1bd5113a54539bf7c8a761e6de7fa: Deleting consensus metadata
I20260928 23:17:07.789656 1491 catalog_manager.cc:4841] TS 59b1bd5113a54539bf7c8a761e6de7fa (127.0.98.1:38547): tablet 646ba3c06bab4a0d87150deabcf7317e (table test-table [id=5ee99d85a09e45f88d966151a2df6efc]) successfully deleted
I20260928 23:17:07.789885 2036 ts_tablet_manager.cc:1905] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:07.790654 2035 ts_tablet_manager.cc:1918] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:07.790699 2035 log.cc:1196] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/646ba3c06bab4a0d87150deabcf7317e
I20260928 23:17:07.790871 2035 ts_tablet_manager.cc:1939] T 646ba3c06bab4a0d87150deabcf7317e P e5ca0f21fbba4734ba6de03258130dfb: Deleting consensus metadata
I20260928 23:17:07.791239 2036 ts_tablet_manager.cc:1918] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:07.791289 2036 log.cc:1196] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/646ba3c06bab4a0d87150deabcf7317e
I20260928 23:17:07.791296 1488 catalog_manager.cc:4841] TS e5ca0f21fbba4734ba6de03258130dfb (127.0.98.3:35853): tablet 646ba3c06bab4a0d87150deabcf7317e (table test-table [id=5ee99d85a09e45f88d966151a2df6efc]) successfully deleted
I20260928 23:17:07.791463 2036 ts_tablet_manager.cc:1939] T 646ba3c06bab4a0d87150deabcf7317e P 05fdee4bfde542ecbbdcf7f2ab0f96da: Deleting consensus metadata
I20260928 23:17:07.791821 1488 catalog_manager.cc:4841] TS 05fdee4bfde542ecbbdcf7f2ab0f96da (127.0.98.2:41273): tablet 646ba3c06bab4a0d87150deabcf7317e (table test-table [id=5ee99d85a09e45f88d966151a2df6efc]) successfully deleted
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637427, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:17:07.803004 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.795800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.0.98.1:38547)
0928 23:17:07.795842 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.795844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.795964 (+ 120us) server_negotiation.cc:407] Connection header received
0928 23:17:07.796020 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.796020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.796030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.796071 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:07.796557 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.797020 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.797725 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.797923 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.797975 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.800560 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:07.800569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:07.800570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:07.800576 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:07.802004 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.802306 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.802309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.802309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.802360 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.802517 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.802518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.802519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.802645 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:07.802767 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.802838 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.802917 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 23:17:07.803203 1702 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50834
I20260928 23:17:07.809283 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:07.804115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.0.98.62:34319)
0928 23:17:07.804160 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:07.804161 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:07.804211 (+ 50us) server_negotiation.cc:407] Connection header received
0928 23:17:07.804268 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:07.804269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:07.804278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:07.804328 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:07.804786 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.805272 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.805912 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:07.806107 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:07.806159 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:07.807110 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:07.807111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:07.807111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:07.807118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:07.808332 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.808614 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.808615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.808615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.808666 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:07.808829 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:07.808829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:07.808830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:07.808955 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:07.809041 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:07.809130 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 23:17:07.809197 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 23:17:07.809544 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35842
I20260928 23:17:07.810144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 1576
I20260928 23:17:07.815199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 1717
I20260928 23:17:07.821786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 1857
I20260928 23:17:07.826076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 1472
2026-09-28T23:17:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6255 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[2072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[2072](info): set up 2 sockets
Sep 28 23:17:07 dist-test-slave-djkr krb5kdc[2072](info): commencing operation
krb5kdc: starting...
W20260928 23:17:08.767094 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:08 dist-test-slave-djkr krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:08.873972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:08.951179 2085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:08.951373 2085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:08.951393 2085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:08.952490 2085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:08.952525 2085 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:17:08.952538 2085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:08.952548 2085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:08.952559 2085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:08.952575 2085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:08.953960 2085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40993
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:08.954167 2085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:08.954402 2085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:08.959587 2093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:08.959702 2091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:08.959759 2085 server_base.cc:1034] running on GCE node
W20260928 23:17:08.959607 2090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:08.960134 2085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:08.960435 2085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:08.961566 2085 hybrid_clock.cc:648] HybridClock initialized: now 1790637428961549 us; error 31 us; skew 500 ppm
Sep 28 23:17:08 dist-test-slave-djkr krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:08.963802 2085 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:08.964200 2085 webserver.cc:460] Webserver started at http://127.0.98.62:39345/ using document root <none> and password file <none>
I20260928 23:17:08.964403 2085 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:08.964452 2085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:08.964570 2085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:08.965593 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "856257ef9fa44ca0919c03eb45b98066"
format_stamp: "Formatted at 2026-09-28 23:17:08 on dist-test-slave-djkr"
server_key: "ea8b856fc8e5dc0c56398a3e1285ba72"
server_key_iv: "71962e81031080bd308ff4cbe046eef4"
server_key_version: "encryptionkey@0"
I20260928 23:17:08.965912 2085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "856257ef9fa44ca0919c03eb45b98066"
format_stamp: "Formatted at 2026-09-28 23:17:08 on dist-test-slave-djkr"
server_key: "ea8b856fc8e5dc0c56398a3e1285ba72"
server_key_iv: "71962e81031080bd308ff4cbe046eef4"
server_key_version: "encryptionkey@0"
I20260928 23:17:08.967099 2085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 23:17:08.967819 2100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:08.967970 2085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:08.968048 2085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "856257ef9fa44ca0919c03eb45b98066"
format_stamp: "Formatted at 2026-09-28 23:17:08 on dist-test-slave-djkr"
server_key: "ea8b856fc8e5dc0c56398a3e1285ba72"
server_key_iv: "71962e81031080bd308ff4cbe046eef4"
server_key_version: "encryptionkey@0"
I20260928 23:17:08.968114 2085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:08.986228 2085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:08.988308 2085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:08.988442 2085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:08.992619 2085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40993
I20260928 23:17:08.992673 2162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40993 every 8 connection(s)
I20260928 23:17:08.992978 2085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:08.993548 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:08.995774 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066: Bootstrap starting.
I20260928 23:17:08.996318 2163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:08.996562 2163 log.cc:824] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:08.997265 2163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066: No bootstrap required, opened a new log
I20260928 23:17:08.998211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 2085
I20260928 23:17:08.998289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:08.998427 392 external_mini_cluster.cc:1428] Setting key c0a1af45e2cff6267c13a01438af9058
I20260928 23:17:08.999631 2163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856257ef9fa44ca0919c03eb45b98066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40993 } }
I20260928 23:17:08.999815 2163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:08.999852 2163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856257ef9fa44ca0919c03eb45b98066, State: Initialized, Role: FOLLOWER
I20260928 23:17:08.999994 2163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856257ef9fa44ca0919c03eb45b98066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40993 } }
I20260928 23:17:09.000074 2163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:09.000097 2163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:09.000144 2163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:09.000813 2163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856257ef9fa44ca0919c03eb45b98066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40993 } }
I20260928 23:17:09.000921 2163 leader_election.cc:302] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 856257ef9fa44ca0919c03eb45b98066; no voters:
I20260928 23:17:09.001062 2163 leader_election.cc:288] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:09.001106 2168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:09.001282 2168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [term 1 LEADER]: Becoming Leader. State: Replica: 856257ef9fa44ca0919c03eb45b98066, State: Running, Role: LEADER
I20260928 23:17:09.001291 2163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:09.001443 2168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856257ef9fa44ca0919c03eb45b98066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40993 } }
I20260928 23:17:09.001847 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "856257ef9fa44ca0919c03eb45b98066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856257ef9fa44ca0919c03eb45b98066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40993 } } }
I20260928 23:17:09.001883 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 856257ef9fa44ca0919c03eb45b98066. Latest consensus state: current_term: 1 leader_uuid: "856257ef9fa44ca0919c03eb45b98066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856257ef9fa44ca0919c03eb45b98066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40993 } } }
I20260928 23:17:09.001924 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:09.002071 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:09.002188 2176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:09.002658 2176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:09.004354 2176 catalog_manager.cc:1347] Generated new cluster ID: b2b4c1de3bce4ea28be61b0baca69a3b
I20260928 23:17:09.004402 2176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:17:09 dist-test-slave-djkr krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:09.009927 2176 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:09.010465 2176 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:09.011276 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:08.998910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.0.98.62:40993)
0928 23:17:08.999135 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:08.999146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:08.999184 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:17:08.999579 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:08.999588 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:08.999653 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:08.999798 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:09.000433 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.001863 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.002802 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.003414 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.003518 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.006193 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:09.006196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:09.006197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:09.006209 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:09.009675 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.010065 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.010069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.010069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.010176 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.010397 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.010398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.010398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.010667 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:09.010826 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.010924 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.011067 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260928 23:17:09.018573 2176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066: Generated new TSK 0
I20260928 23:17:09.018733 2176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:17:09.042538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40993
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:09.120419 2189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:09.120611 2189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:09.120635 2189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:09.121759 2189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:09.121800 2189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:09.121819 2189 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:17:09.121829 2189 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:17:09.121845 2189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:09.123256 2189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:09.123466 2189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:09.123687 2189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:09.128315 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:09.128299 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:09.128299 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:09.128468 2189 server_base.cc:1034] running on GCE node
I20260928 23:17:09.128729 2189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:09.128957 2189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:09.130095 2189 hybrid_clock.cc:648] HybridClock initialized: now 1790637429130081 us; error 29 us; skew 500 ppm
Sep 28 23:17:09 dist-test-slave-djkr krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:09.132344 2189 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:17:09.132733 2189 webserver.cc:460] Webserver started at http://127.0.98.1:40745/ using document root <none> and password file <none>
I20260928 23:17:09.132921 2189 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:09.132967 2189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:09.133088 2189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:09.134078 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "3518f3b95b774723a329e691fa929aa5"
format_stamp: "Formatted at 2026-09-28 23:17:09 on dist-test-slave-djkr"
server_key: "91717bcd7923770603406b5d26057290"
server_key_iv: "5e1c0e9a51ad5f6758705e914422234b"
server_key_version: "encryptionkey@0"
I20260928 23:17:09.134399 2189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3518f3b95b774723a329e691fa929aa5"
format_stamp: "Formatted at 2026-09-28 23:17:09 on dist-test-slave-djkr"
server_key: "91717bcd7923770603406b5d26057290"
server_key_iv: "5e1c0e9a51ad5f6758705e914422234b"
server_key_version: "encryptionkey@0"
I20260928 23:17:09.135684 2189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:17:09.136435 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.136586 2189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:09.136677 2189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "3518f3b95b774723a329e691fa929aa5"
format_stamp: "Formatted at 2026-09-28 23:17:09 on dist-test-slave-djkr"
server_key: "91717bcd7923770603406b5d26057290"
server_key_iv: "5e1c0e9a51ad5f6758705e914422234b"
server_key_version: "encryptionkey@0"
I20260928 23:17:09.136746 2189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:09.144084 2189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:09.145807 2189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:09.145927 2189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:09.146378 2189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:09.146420 2189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.146447 2189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:09.146476 2189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.152415 2189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38843
I20260928 23:17:09.152467 2319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38843 every 8 connection(s)
Sep 28 23:17:09 dist-test-slave-djkr krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:09.152745 2189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:09.155408 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.146828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35945 (local address 127.0.98.62:40993)
0928 23:17:09.146892 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.146894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.147064 (+ 170us) server_negotiation.cc:407] Connection header received
0928 23:17:09.147518 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.147519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.147532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.147624 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:09.148353 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.149103 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.150525 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.150834 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.150909 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.153289 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:09.153290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:09.153291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:09.153298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:09.154368 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.154731 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.154732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.154732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.154791 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.154966 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.154967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.154967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.155057 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:09.155099 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.155281 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.155346 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:17:09.155606 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.146778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40993 (local address 127.0.98.1:35945)
0928 23:17:09.146948 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.146986 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.147435 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.147703 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.147707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.148002 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:09.148256 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.148263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.149193 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.149194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.150439 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.150442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.150908 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.150909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.150999 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.151485 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:09.151493 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:09.153152 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:09.154432 (+ 1280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.154437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.154438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.154661 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.154840 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.154841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.154841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.154930 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.155096 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:09.155098 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:09.155197 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.155268 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.155384 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260928 23:17:09.156049 2320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40993
I20260928 23:17:09.156123 2320 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:09.156283 2320 heartbeater.cc:511] Master 127.0.98.62:40993 requested a full tablet report, sending...
I20260928 23:17:09.156622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 2189
I20260928 23:17:09.156718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:09.156744 2117 ts_manager.cc:194] Registered new tserver with Master: 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843)
I20260928 23:17:09.156895 392 external_mini_cluster.cc:1428] Setting key bb5b51e753095d2c296a41770c2f58ba
I20260928 23:17:09.157522 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35945
I20260928 23:17:09.162426 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.156957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40993 (local address 127.0.98.1:45699)
0928 23:17:09.157087 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.157096 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.157165 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.157311 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.157311 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.157458 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:09.157501 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.157502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.158288 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.158289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.159000 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.159002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.159444 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.159445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.159495 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.159894 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:09.159898 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:09.160135 (+ 237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:09.161626 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.161626 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.161627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.161773 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.161955 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.161955 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.161955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.161998 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.162203 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:09.162204 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:09.162234 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.162277 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.162360 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260928 23:17:09.162454 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.157015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45699 (local address 127.0.98.62:40993)
0928 23:17:09.157049 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.157050 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.157142 (+ 92us) server_negotiation.cc:407] Connection header received
0928 23:17:09.157227 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.157228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.157240 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.157265 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:09.157583 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.158190 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.159074 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.159379 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.159448 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.160211 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:09.160212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:09.160212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:09.160219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:09.161553 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.161835 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.161837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.161837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.161893 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.162034 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.162034 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.162034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.162152 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:09.162216 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.162304 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.162371 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:17:09.180392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40993
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:09.257766 2329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:09.257959 2329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:09.257980 2329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:09.259073 2329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:09.259114 2329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:09.259133 2329 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:17:09.259156 2329 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:17:09.259182 2329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:09.260600 2329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:09.260800 2329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:09.261000 2329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:09.265605 2337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:09.265708 2329 server_base.cc:1034] running on GCE node
W20260928 23:17:09.265623 2334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:09.265796 2335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:09.265969 2329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:09.266192 2329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:09.267367 2329 hybrid_clock.cc:648] HybridClock initialized: now 1790637429267352 us; error 28 us; skew 500 ppm
Sep 28 23:17:09 dist-test-slave-djkr krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:09.269695 2329 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:17:09.270123 2329 webserver.cc:460] Webserver started at http://127.0.98.2:33619/ using document root <none> and password file <none>
I20260928 23:17:09.270320 2329 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:09.270366 2329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:09.270484 2329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:09.271490 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "1bf932d184934198b8bf59c3efbabfe9"
format_stamp: "Formatted at 2026-09-28 23:17:09 on dist-test-slave-djkr"
server_key: "d425a9e6fc2e52a666821783735ac76b"
server_key_iv: "18dcf64c3a577b4b5a20511fdc10a486"
server_key_version: "encryptionkey@0"
I20260928 23:17:09.271879 2329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1bf932d184934198b8bf59c3efbabfe9"
format_stamp: "Formatted at 2026-09-28 23:17:09 on dist-test-slave-djkr"
server_key: "d425a9e6fc2e52a666821783735ac76b"
server_key_iv: "18dcf64c3a577b4b5a20511fdc10a486"
server_key_version: "encryptionkey@0"
I20260928 23:17:09.273042 2329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:09.273751 2344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.273916 2329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:09.273994 2329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "1bf932d184934198b8bf59c3efbabfe9"
format_stamp: "Formatted at 2026-09-28 23:17:09 on dist-test-slave-djkr"
server_key: "d425a9e6fc2e52a666821783735ac76b"
server_key_iv: "18dcf64c3a577b4b5a20511fdc10a486"
server_key_version: "encryptionkey@0"
I20260928 23:17:09.274065 2329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:09.288751 2329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:09.290691 2329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:09.290820 2329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:09.291361 2329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:09.291411 2329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.291440 2329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:09.291455 2329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:17:09 dist-test-slave-djkr krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:09.298300 2329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39121
I20260928 23:17:09.298377 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39121 every 8 connection(s)
I20260928 23:17:09.298660 2329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:09.300663 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.291760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50319 (local address 127.0.98.62:40993)
0928 23:17:09.291821 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.291824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.292056 (+ 232us) server_negotiation.cc:407] Connection header received
0928 23:17:09.292699 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.292700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.292712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.292784 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:09.293582 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.294091 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.295563 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.295792 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.295858 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.298136 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:09.298137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:09.298137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:09.298143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:09.299501 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.299903 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.299904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.299905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.299953 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.300155 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.300156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.300157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.300289 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:09.300352 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.300547 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.300603 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:17:09.300827 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.291760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40993 (local address 127.0.98.2:50319)
0928 23:17:09.291915 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.291962 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.292578 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.292872 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.292877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.293206 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:09.293485 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.293492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.294160 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.294161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.295456 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.295459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.295887 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.295888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.296003 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.296479 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:09.296488 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:09.298043 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:09.299599 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.299604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.299606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.299835 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.299995 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.299996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.299996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.300081 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.300349 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:09.300353 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:09.300457 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.300533 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.300640 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260928 23:17:09.301257 2460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40993
I20260928 23:17:09.301340 2460 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:09.301497 2460 heartbeater.cc:511] Master 127.0.98.62:40993 requested a full tablet report, sending...
I20260928 23:17:09.301841 2116 ts_manager.cc:194] Registered new tserver with Master: 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121)
I20260928 23:17:09.302320 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50319
I20260928 23:17:09.304441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 2329
I20260928 23:17:09.304533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:09.304723 392 external_mini_cluster.cc:1428] Setting key fe0f83ccd604788c4ca83da95970ed41
I20260928 23:17:09.307183 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.302169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54309 (local address 127.0.98.62:40993)
0928 23:17:09.302200 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.302202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.302323 (+ 121us) server_negotiation.cc:407] Connection header received
0928 23:17:09.302365 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.302366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.302374 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.302425 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:09.302773 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.303302 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.304013 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.304205 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.304262 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.305103 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:09.305104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:09.305105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:09.305111 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:09.306240 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.306512 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.306513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.306514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.306563 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.306712 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.306713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.306713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.306846 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:09.306899 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.307036 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.307094 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 23:17:09.307196 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.302104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40993 (local address 127.0.98.2:54309)
0928 23:17:09.302255 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.302265 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.302323 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.302466 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.302467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.302646 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:09.302701 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.302702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.303381 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.303382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.303948 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.303950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.304266 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.304266 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.304337 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.304711 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:09.304715 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:09.305038 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:09.306309 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.306310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.306310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.306458 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.306613 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.306613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.306614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.306666 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.306900 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:09.306901 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:09.306941 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.307000 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.307094 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:17:09.327652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40993
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:09.404834 2469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:09.405027 2469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:09.405059 2469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:09.406173 2469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:09.406211 2469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:09.406239 2469 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:17:09.406252 2469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:17:09.406275 2469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:09.407728 2469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:09.407958 2469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:09.408186 2469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:09.412739 2475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:09.412748 2477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:09.412799 2469 server_base.cc:1034] running on GCE node
W20260928 23:17:09.412768 2474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:09.413131 2469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:09.413328 2469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:09.414481 2469 hybrid_clock.cc:648] HybridClock initialized: now 1790637429414467 us; error 27 us; skew 500 ppm
Sep 28 23:17:09 dist-test-slave-djkr krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:09.416846 2469 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:17:09.417280 2469 webserver.cc:460] Webserver started at http://127.0.98.3:36579/ using document root <none> and password file <none>
I20260928 23:17:09.417483 2469 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:09.417529 2469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:09.417650 2469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:09.418638 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "74f9a13feedf4d3ba37d0d672f5c38f1"
format_stamp: "Formatted at 2026-09-28 23:17:09 on dist-test-slave-djkr"
server_key: "8126abd8645d18d7e6157b1c3769063d"
server_key_iv: "6d4a694ea0c32937f9f3430288f42850"
server_key_version: "encryptionkey@0"
I20260928 23:17:09.418953 2469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "74f9a13feedf4d3ba37d0d672f5c38f1"
format_stamp: "Formatted at 2026-09-28 23:17:09 on dist-test-slave-djkr"
server_key: "8126abd8645d18d7e6157b1c3769063d"
server_key_iv: "6d4a694ea0c32937f9f3430288f42850"
server_key_version: "encryptionkey@0"
I20260928 23:17:09.420272 2469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:09.420925 2484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.421066 2469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:09.421144 2469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "74f9a13feedf4d3ba37d0d672f5c38f1"
format_stamp: "Formatted at 2026-09-28 23:17:09 on dist-test-slave-djkr"
server_key: "8126abd8645d18d7e6157b1c3769063d"
server_key_iv: "6d4a694ea0c32937f9f3430288f42850"
server_key_version: "encryptionkey@0"
I20260928 23:17:09.421201 2469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:09.434026 2469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:09.435765 2469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:09.435894 2469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:09.436363 2469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:09.436403 2469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.436432 2469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:09.436455 2469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:17:09 dist-test-slave-djkr krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:09.442925 2469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44025
I20260928 23:17:09.443037 2599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44025 every 8 connection(s)
I20260928 23:17:09.443331 2469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:09.445506 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.436744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46191 (local address 127.0.98.62:40993)
0928 23:17:09.436797 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.436799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.437007 (+ 208us) server_negotiation.cc:407] Connection header received
0928 23:17:09.437484 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.437486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.437498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.437547 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:09.438283 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.439034 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.440105 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.440420 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.440500 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.442754 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:09.442759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:09.442760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:09.442768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:09.444287 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.444644 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.444646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.444646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.444701 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.444915 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.444916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.444916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.445022 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:09.445076 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.445342 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.445418 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:17:09.445652 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.436756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40993 (local address 127.0.98.3:46191)
0928 23:17:09.436896 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.436929 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.437392 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.437612 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.437615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.437911 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:09.438178 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.438185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.439130 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.439130 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.440036 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.440039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.440519 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.440520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.440653 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.441113 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:09.441123 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:09.442633 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:09.444363 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.444369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.444371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.444573 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.444759 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.444760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.444760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.444860 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.445074 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:09.445080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:09.445219 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.445306 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.445451 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260928 23:17:09.446092 2600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40993
I20260928 23:17:09.446166 2600 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:09.446314 2600 heartbeater.cc:511] Master 127.0.98.62:40993 requested a full tablet report, sending...
I20260928 23:17:09.446638 2116 ts_manager.cc:194] Registered new tserver with Master: 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3:44025)
I20260928 23:17:09.447099 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46191
I20260928 23:17:09.451632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 2469
I20260928 23:17:09.451728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:09.451938 392 external_mini_cluster.cc:1428] Setting key ab0c81f24e7732fdcc3f51361d432c17
I20260928 23:17:09.452154 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.446947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40993 (local address 127.0.98.3:33237)
0928 23:17:09.447280 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.447291 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.447348 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.447886 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.447886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.448019 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:09.448072 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.448073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.448626 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.448627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.449121 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.449123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.449401 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.449402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.449451 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.449824 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:09.449829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:09.450090 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:09.451213 (+ 1123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.451214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.451214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.451365 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.451516 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.451516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.451516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.451585 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.451853 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:09.451854 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:09.451896 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.451971 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.452060 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":77,"threads_started":1}
I20260928 23:17:09.452174 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.447002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33237 (local address 127.0.98.62:40993)
0928 23:17:09.447042 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.447043 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.447346 (+ 303us) server_negotiation.cc:407] Connection header received
0928 23:17:09.447816 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.447817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.447825 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.447872 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:09.448121 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.448557 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.449180 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.449351 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.449395 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.450153 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:09.450153 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:09.450154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:09.450159 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:09.451148 (+ 989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.451416 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.451416 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.451416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.451465 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.451632 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.451633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.451633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.451747 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:09.451822 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.452023 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.452094 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:17:09.452922 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:17:09 dist-test-slave-djkr krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:17:09 dist-test-slave-djkr krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:09.473373 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.467006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.0.98.62:40993)
0928 23:17:09.467077 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.467079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.467087 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:09.467109 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.467110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.467118 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.467166 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:09.467679 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.468178 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.468934 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.469119 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.469173 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.471158 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:09.471159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:09.471159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:09.471165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:09.472466 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.472783 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.472783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.472784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.472831 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.472961 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.472961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.472961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.473069 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:09.473150 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.473236 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.473292 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:17:09.474833 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:09.476418 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:09.482646 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.478546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39121 (local address 127.0.0.1:48916)
0928 23:17:09.478697 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.478734 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.478793 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.479287 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.479288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.479291 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:09.479336 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.479338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.480265 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.480266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.481522 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.481524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.482383 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.482384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.482456 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.482469 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.482533 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.482578 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":59,"threads_started":1}
I20260928 23:17:09.482744 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.478725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38843 (local address 127.0.0.1:49456)
0928 23:17:09.478905 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.478915 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.478954 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.479268 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.479268 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.479270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:09.479305 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.479306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.480607 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.480608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.481624 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.481626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.482482 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.482483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.482544 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.482548 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.482601 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.482647 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260928 23:17:09.482901 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.478947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.0.98.1:38843)
0928 23:17:09.479112 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.479117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.479127 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:09.479165 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.479167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.479180 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.479260 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:09.479381 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.480517 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.481687 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.482390 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.482475 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.482540 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.482616 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.482653 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260928 23:17:09.483129 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.478956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.0.98.2:39121)
0928 23:17:09.479138 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.479142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.479150 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:09.479181 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.479183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.479195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.479417 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:09.479484 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.480188 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.481591 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.482299 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.482376 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.482441 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.482990 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.483032 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20260928 23:17:09.483467 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.478620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.0.98.3:44025)
0928 23:17:09.478770 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.478775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.478954 (+ 179us) server_negotiation.cc:407] Connection header received
0928 23:17:09.478997 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.478999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.479015 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.479085 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:09.479398 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.480525 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.482414 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.483126 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.483188 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.483245 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.483329 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.483376 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260928 23:17:09.483521 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2025fa116b4613a59492920a09c522 (DEFAULT_TABLE table=test-table [id=20516b13204e4aa3b4b7b5245997cc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:09.483732 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2025fa116b4613a59492920a09c522 (DEFAULT_TABLE table=test-table [id=20516b13204e4aa3b4b7b5245997cc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:09.483423 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.478546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44025 (local address 127.0.0.1:58946)
0928 23:17:09.478836 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.478845 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.478898 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.479161 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.479164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.479166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:09.479212 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.479213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.481691 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.481691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.482325 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.482326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.483206 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.483207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.483268 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.483271 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.483317 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.483361 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1,"thread_start_us":49,"threads_started":1}
I20260928 23:17:09.483917 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2025fa116b4613a59492920a09c522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:09.484017 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2025fa116b4613a59492920a09c522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:09.485529 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2025fa116b4613a59492920a09c522 (DEFAULT_TABLE table=test-table [id=20516b13204e4aa3b4b7b5245997cc1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:09.485826 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2025fa116b4613a59492920a09c522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:09.487421 2621 tablet_bootstrap.cc:492] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: Bootstrap starting.
I20260928 23:17:09.487962 2621 tablet_bootstrap.cc:654] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:09.488219 2621 log.cc:824] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:09.488435 2622 tablet_bootstrap.cc:492] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: Bootstrap starting.
I20260928 23:17:09.488624 2620 tablet_bootstrap.cc:492] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: Bootstrap starting.
I20260928 23:17:09.488905 2621 tablet_bootstrap.cc:492] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: No bootstrap required, opened a new log
I20260928 23:17:09.488960 2621 ts_tablet_manager.cc:1397] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:09.489204 2620 tablet_bootstrap.cc:654] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:09.489274 2622 tablet_bootstrap.cc:654] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:09.489533 2620 log.cc:824] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:09.489554 2622 log.cc:824] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:09.490219 2622 tablet_bootstrap.cc:492] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: No bootstrap required, opened a new log
I20260928 23:17:09.490271 2622 ts_tablet_manager.cc:1397] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:09.490494 2620 tablet_bootstrap.cc:492] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: No bootstrap required, opened a new log
I20260928 23:17:09.490593 2620 ts_tablet_manager.cc:1397] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:17:09.490756 2621 raft_consensus.cc:348] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.490911 2621 raft_consensus.cc:374] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:09.490947 2621 raft_consensus.cc:729] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf932d184934198b8bf59c3efbabfe9, State: Initialized, Role: FOLLOWER
I20260928 23:17:09.491053 2621 consensus_queue.cc:260] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.491302 2621 ts_tablet_manager.cc:1428] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:17:09.491334 2460 heartbeater.cc:503] Master 127.0.98.62:40993 was elected leader, sending a full tablet report...
I20260928 23:17:09.491962 2622 raft_consensus.cc:348] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.492113 2622 raft_consensus.cc:374] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:09.492153 2622 raft_consensus.cc:729] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f9a13feedf4d3ba37d0d672f5c38f1, State: Initialized, Role: FOLLOWER
I20260928 23:17:09.492257 2622 consensus_queue.cc:260] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.492502 2622 ts_tablet_manager.cc:1428] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:17:09.492506 2600 heartbeater.cc:503] Master 127.0.98.62:40993 was elected leader, sending a full tablet report...
I20260928 23:17:09.492725 2620 raft_consensus.cc:348] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.492874 2620 raft_consensus.cc:374] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:09.492918 2620 raft_consensus.cc:729] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3518f3b95b774723a329e691fa929aa5, State: Initialized, Role: FOLLOWER
I20260928 23:17:09.493023 2620 consensus_queue.cc:260] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.493257 2620 ts_tablet_manager.cc:1428] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:17:09.493279 2320 heartbeater.cc:503] Master 127.0.98.62:40993 was elected leader, sending a full tablet report...
I20260928 23:17:09.502372 2626 raft_consensus.cc:482] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:09.502466 2626 raft_consensus.cc:504] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.502758 2626 leader_election.cc:288] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843), 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3:44025)
I20260928 23:17:09.505414 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.502912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44025 (local address 127.0.98.2:42037)
0928 23:17:09.502967 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.502979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.503015 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.503158 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.503159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.503161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:09.503224 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.503225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.503917 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.503918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.504657 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.504659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.505164 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.505164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.505250 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.505254 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.505297 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.505338 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 23:17:09.505406 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.502973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42037 (local address 127.0.98.3:44025)
0928 23:17:09.503048 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.503050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.503057 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:09.503079 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.503080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.503087 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.503139 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:09.503293 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.503840 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.504736 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.505097 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.505156 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.505167 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.505307 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.505340 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:17:09.505684 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.502974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56153 (local address 127.0.98.1:38843)
0928 23:17:09.503018 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.503020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.503264 (+ 244us) server_negotiation.cc:407] Connection header received
0928 23:17:09.503326 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.503327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.503337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.503376 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:09.503499 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.504104 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.504983 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.505357 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.505413 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.505422 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.505584 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.505628 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:17:09.505705 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.502914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38843 (local address 127.0.98.2:56153)
0928 23:17:09.503205 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.503214 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.503261 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.503387 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.503388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.503389 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:09.503449 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.503451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.504198 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.504199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.504904 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.504906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.505443 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.505444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.505519 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.505523 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.505577 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.505613 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":150,"thread_start_us":81,"threads_started":1}
I20260928 23:17:09.506002 2551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2025fa116b4613a59492920a09c522" candidate_uuid: "1bf932d184934198b8bf59c3efbabfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" is_pre_election: true
I20260928 23:17:09.506070 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2025fa116b4613a59492920a09c522" candidate_uuid: "1bf932d184934198b8bf59c3efbabfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518f3b95b774723a329e691fa929aa5" is_pre_election: true
I20260928 23:17:09.506150 2551 raft_consensus.cc:2454] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bf932d184934198b8bf59c3efbabfe9 in term 0.
I20260928 23:17:09.506201 2274 raft_consensus.cc:2454] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bf932d184934198b8bf59c3efbabfe9 in term 0.
I20260928 23:17:09.506366 2347 leader_election.cc:302] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf932d184934198b8bf59c3efbabfe9, 74f9a13feedf4d3ba37d0d672f5c38f1; no voters:
I20260928 23:17:09.506495 2626 raft_consensus.cc:2781] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:09.506546 2626 raft_consensus.cc:482] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:09.506562 2626 raft_consensus.cc:3037] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:09.507179 2626 raft_consensus.cc:504] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.507284 2626 leader_election.cc:288] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [CANDIDATE]: Term 1 election: Requested vote from peers 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843), 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3:44025)
I20260928 23:17:09.507421 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2025fa116b4613a59492920a09c522" candidate_uuid: "1bf932d184934198b8bf59c3efbabfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518f3b95b774723a329e691fa929aa5"
I20260928 23:17:09.507433 2551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2025fa116b4613a59492920a09c522" candidate_uuid: "1bf932d184934198b8bf59c3efbabfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1"
I20260928 23:17:09.507494 2274 raft_consensus.cc:3037] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:09.507503 2551 raft_consensus.cc:3037] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:09.508181 2274 raft_consensus.cc:2454] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf932d184934198b8bf59c3efbabfe9 in term 1.
I20260928 23:17:09.508184 2551 raft_consensus.cc:2454] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf932d184934198b8bf59c3efbabfe9 in term 1.
I20260928 23:17:09.508316 2347 leader_election.cc:302] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf932d184934198b8bf59c3efbabfe9, 74f9a13feedf4d3ba37d0d672f5c38f1; no voters:
I20260928 23:17:09.508425 2626 raft_consensus.cc:2781] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:09.508563 2626 raft_consensus.cc:686] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 LEADER]: Becoming Leader. State: Replica: 1bf932d184934198b8bf59c3efbabfe9, State: Running, Role: LEADER
I20260928 23:17:09.508641 2626 consensus_queue.cc:237] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.509356 2116 catalog_manager.cc:5495] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1bf932d184934198b8bf59c3efbabfe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } health_report { overall_health: UNKNOWN } } }
Sep 28 23:17:09 dist-test-slave-djkr krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:09.533931 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.528102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40993 (local address 127.0.0.1:55386)
0928 23:17:09.528259 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.528270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.528334 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.528463 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.528464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.528742 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:09.528787 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.528789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.529414 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.529415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.529936 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.529937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.530204 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.530205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.530252 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.530688 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:09.530692 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:09.531934 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:09.533088 (+ 1154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.533093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.533095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.533293 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.533432 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:09.533432 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:09.533432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:09.533482 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:09.533645 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:09.533649 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:09.533685 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.533730 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.533811 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260928 23:17:09.533939 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.528210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.0.98.62:40993)
0928 23:17:09.528280 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.528282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.528326 (+ 44us) server_negotiation.cc:407] Connection header received
0928 23:17:09.528389 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.528389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.528398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.528448 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:09.528878 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.529347 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.529991 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.530157 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.530203 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.532003 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:09.532004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:09.532004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:09.532010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:09.533028 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.533347 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.533347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.533348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.533393 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:09.533519 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:09.533519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:09.533519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:09.533604 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:09.533647 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.533751 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.533851 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":47}
I20260928 23:17:09.535646 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55386:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:17:09.535815 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:09.537058 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 7a173da6826d4806b2c1cd31964c1654 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a7ea3fc674e2424cae679593692a30f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:17:09.537051 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 7a173da6826d4806b2c1cd31964c1654 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a7ea3fc674e2424cae679593692a30f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:17:09.537190 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a173da6826d4806b2c1cd31964c1654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:09.537205 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a173da6826d4806b2c1cd31964c1654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:09.537204 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 7a173da6826d4806b2c1cd31964c1654 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a7ea3fc674e2424cae679593692a30f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:17:09.537328 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a173da6826d4806b2c1cd31964c1654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:09.538190 2622 tablet_bootstrap.cc:492] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: Bootstrap starting.
I20260928 23:17:09.538653 2622 tablet_bootstrap.cc:654] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:09.538666 2621 tablet_bootstrap.cc:492] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: Bootstrap starting.
I20260928 23:17:09.538684 2620 tablet_bootstrap.cc:492] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: Bootstrap starting.
I20260928 23:17:09.539028 2621 tablet_bootstrap.cc:654] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:09.539330 2620 tablet_bootstrap.cc:654] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:09.539340 2622 tablet_bootstrap.cc:492] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: No bootstrap required, opened a new log
I20260928 23:17:09.539386 2622 ts_tablet_manager.cc:1397] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:09.539534 2622 raft_consensus.cc:348] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.539630 2622 raft_consensus.cc:374] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:09.539651 2622 raft_consensus.cc:729] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f9a13feedf4d3ba37d0d672f5c38f1, State: Initialized, Role: FOLLOWER
I20260928 23:17:09.539655 2621 tablet_bootstrap.cc:492] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: No bootstrap required, opened a new log
I20260928 23:17:09.539697 2621 ts_tablet_manager.cc:1397] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:09.539690 2622 consensus_queue.cc:260] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.539770 2622 ts_tablet_manager.cc:1428] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:09.539821 2621 raft_consensus.cc:348] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.539897 2621 raft_consensus.cc:374] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:09.539916 2621 raft_consensus.cc:729] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf932d184934198b8bf59c3efbabfe9, State: Initialized, Role: FOLLOWER
I20260928 23:17:09.539951 2621 consensus_queue.cc:260] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.539917 2627 tablet_replica.cc:439] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } } }
I20260928 23:17:09.539997 2627 tablet_replica.cc:442] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:09.540028 2621 ts_tablet_manager.cc:1428] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.540143 2626 tablet_replica.cc:439] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } } }
I20260928 23:17:09.540239 2626 tablet_replica.cc:442] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:09.540714 2620 tablet_bootstrap.cc:492] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: No bootstrap required, opened a new log
I20260928 23:17:09.540755 2620 ts_tablet_manager.cc:1397] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:09.540867 2620 raft_consensus.cc:348] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.540925 2620 raft_consensus.cc:374] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:09.540937 2620 raft_consensus.cc:729] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3518f3b95b774723a329e691fa929aa5, State: Initialized, Role: FOLLOWER
I20260928 23:17:09.540966 2620 consensus_queue.cc:260] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.541043 2620 ts_tablet_manager.cc:1428] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.541102 2628 tablet_replica.cc:439] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } } }
I20260928 23:17:09.541170 2628 tablet_replica.cc:442] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 23:17:09.549315 2461 tablet.cc:2369] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:09.554060 2627 raft_consensus.cc:482] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:09.554121 2627 raft_consensus.cc:504] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.554389 2627 leader_election.cc:288] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843), 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121)
I20260928 23:17:09.557978 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.554702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50547 (local address 127.0.98.2:39121)
0928 23:17:09.554736 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.554738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.554796 (+ 58us) server_negotiation.cc:407] Connection header received
0928 23:17:09.554836 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.554837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.554845 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.554894 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:09.554999 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.555708 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.557092 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.557631 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.557701 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.557710 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.557883 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.557917 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:17:09.557987 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.554573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40685 (local address 127.0.98.1:38843)
0928 23:17:09.554616 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.554618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.554626 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:09.554690 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.554690 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.554699 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.554757 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:09.554903 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.555665 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.557101 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.557632 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.557701 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.557710 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.557897 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.557933 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:17:09.557984 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.554523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38843 (local address 127.0.98.3:40685)
0928 23:17:09.554566 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.554576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.554614 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.554763 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.554764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.554766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:09.554829 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.554831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.555731 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.555732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.557024 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.557027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.557714 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.557715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.557801 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.557807 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.557889 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.557918 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":92}
I20260928 23:17:09.557999 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.554611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39121 (local address 127.0.98.3:50547)
0928 23:17:09.554736 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.554744 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.554788 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.554896 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.554896 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.554898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:09.554952 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.554953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.555777 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.555777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.557024 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.557027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.557729 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.557730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.557801 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.557808 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.557865 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.557906 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":22,"thread_start_us":89,"threads_started":1}
I20260928 23:17:09.558210 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a173da6826d4806b2c1cd31964c1654" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518f3b95b774723a329e691fa929aa5" is_pre_election: true
I20260928 23:17:09.558279 2274 raft_consensus.cc:2454] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 0.
I20260928 23:17:09.558434 2485 leader_election.cc:302] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3518f3b95b774723a329e691fa929aa5, 74f9a13feedf4d3ba37d0d672f5c38f1; no voters:
I20260928 23:17:09.558442 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a173da6826d4806b2c1cd31964c1654" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf932d184934198b8bf59c3efbabfe9" is_pre_election: true
I20260928 23:17:09.558516 2414 raft_consensus.cc:2454] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 0.
I20260928 23:17:09.558564 2627 raft_consensus.cc:2781] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:09.558604 2627 raft_consensus.cc:482] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:09.558629 2627 raft_consensus.cc:3037] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:09.559250 2627 raft_consensus.cc:504] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.559362 2627 leader_election.cc:288] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 election: Requested vote from peers 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843), 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121)
I20260928 23:17:09.559504 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a173da6826d4806b2c1cd31964c1654" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf932d184934198b8bf59c3efbabfe9"
I20260928 23:17:09.559527 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a173da6826d4806b2c1cd31964c1654" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518f3b95b774723a329e691fa929aa5"
I20260928 23:17:09.559607 2414 raft_consensus.cc:3037] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:09.559608 2274 raft_consensus.cc:3037] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:09.560300 2274 raft_consensus.cc:2454] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 1.
I20260928 23:17:09.560341 2414 raft_consensus.cc:2454] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 1.
I20260928 23:17:09.560467 2485 leader_election.cc:302] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3518f3b95b774723a329e691fa929aa5, 74f9a13feedf4d3ba37d0d672f5c38f1; no voters:
I20260928 23:17:09.560550 2627 raft_consensus.cc:2781] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:09.560674 2627 raft_consensus.cc:686] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 LEADER]: Becoming Leader. State: Replica: 74f9a13feedf4d3ba37d0d672f5c38f1, State: Running, Role: LEADER
I20260928 23:17:09.560770 2627 consensus_queue.cc:237] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.561178 2642 tablet_replica.cc:439] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: TxnStatusTablet state changed. Reason: New leader 74f9a13feedf4d3ba37d0d672f5c38f1. Latest consensus state: current_term: 1 leader_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } } }
I20260928 23:17:09.561260 2642 tablet_replica.cc:442] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:17:09.561486 2644 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:17:09.561488 2117 catalog_manager.cc:5495] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } health_report { overall_health: HEALTHY } } }
I20260928 23:17:09.561599 2644 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:17:09.569721 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.567603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.0.98.3:44025)
0928 23:17:09.567658 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.567660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.567668 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:09.567723 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.567724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.567740 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.567798 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:09.567891 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.568346 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.569032 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.569225 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.569275 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.569437 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:09.569526 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:09.569590 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.569639 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.569658 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:17:09.569742 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.567520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44025 (local address 127.0.0.1:58948)
0928 23:17:09.567597 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.567607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.567649 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.567801 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.567802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.567803 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:17:09.567846 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.567848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.568416 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.568417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.568966 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.568967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.569280 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.569281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.569343 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.569355 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:17:09.569578 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:17:09.569585 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.569634 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.569679 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 23:17:09.571460 2414 raft_consensus.cc:1261] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Refusing update from remote peer 74f9a13feedf4d3ba37d0d672f5c38f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:09.571478 2274 raft_consensus.cc:1261] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Refusing update from remote peer 74f9a13feedf4d3ba37d0d672f5c38f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:09.571761 2627 consensus_queue.cc:1035] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:09.571872 2642 consensus_queue.cc:1035] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:09.572911 2626 tablet_replica.cc:439] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: TxnStatusTablet state changed. Reason: New leader 74f9a13feedf4d3ba37d0d672f5c38f1. Latest consensus state: current_term: 1 leader_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } } }
I20260928 23:17:09.572996 2626 tablet_replica.cc:442] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:09.572968 2628 tablet_replica.cc:439] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: TxnStatusTablet state changed. Reason: New leader 74f9a13feedf4d3ba37d0d672f5c38f1. Latest consensus state: current_term: 1 leader_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } } }
I20260928 23:17:09.573063 2628 tablet_replica.cc:442] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:09.573932 2642 tablet_replica.cc:439] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } } }
I20260928 23:17:09.574007 2642 tablet_replica.cc:442] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:17:09.573964 2627 tablet_replica.cc:439] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } } }
I20260928 23:17:09.574040 2627 tablet_replica.cc:442] T 7a173da6826d4806b2c1cd31964c1654 P 74f9a13feedf4d3ba37d0d672f5c38f1: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:17:09.574244 2626 tablet_replica.cc:439] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } } }
I20260928 23:17:09.574316 2626 tablet_replica.cc:442] T 7a173da6826d4806b2c1cd31964c1654 P 1bf932d184934198b8bf59c3efbabfe9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:09.574383 2648 mvcc.cc:205] Tried to move back new op lower bound from 7334450911522590720 to 7334450911481561088. Current Snapshot: MvccSnapshot[applied={T|T < 7334450911522590720}]
I20260928 23:17:09.574600 2628 tablet_replica.cc:439] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } } }
I20260928 23:17:09.574677 2628 tablet_replica.cc:442] T 7a173da6826d4806b2c1cd31964c1654 P 3518f3b95b774723a329e691fa929aa5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:09.574757 2649 mvcc.cc:205] Tried to move back new op lower bound from 7334450911522590720 to 7334450911481561088. Current Snapshot: MvccSnapshot[applied={T|T < 7334450911522590720}]
I20260928 23:17:09.579563 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.576995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.0.98.2:39121)
0928 23:17:09.577029 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.577031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.577040 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:09.577112 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.577112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.577125 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.577170 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:09.577304 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.577785 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.578870 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.579086 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.579146 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.579296 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:09.579376 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:09.579425 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.579484 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.579500 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:17:09.584746 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.581741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44025 (local address 127.0.98.2:48213)
0928 23:17:09.581814 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.581825 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.581869 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.582032 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.582033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.582034 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:17:09.582101 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.582102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.582924 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.582925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.583801 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.583803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.584254 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.584255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.584318 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.584333 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:17:09.584597 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:17:09.584599 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.584639 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.584678 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 23:17:09.584765 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.581832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48213 (local address 127.0.98.3:44025)
0928 23:17:09.581862 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.581863 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.581871 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:09.581979 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.581980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.581988 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.582029 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:09.582174 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.582835 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.583886 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.584172 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.584226 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.584496 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:09.584529 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:09.584574 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.584667 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.584684 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:17:09.587014 2274 raft_consensus.cc:1261] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Refusing update from remote peer 1bf932d184934198b8bf59c3efbabfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:09.587169 2551 raft_consensus.cc:1261] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Refusing update from remote peer 1bf932d184934198b8bf59c3efbabfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:09.587292 2626 consensus_queue.cc:1035] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:09.587440 2626 consensus_queue.cc:1035] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:09.589046 2647 mvcc.cc:205] Tried to move back new op lower bound from 7334450911586807808 to 7334450911267946496. Current Snapshot: MvccSnapshot[applied={T|T < 7334450911586807808}]
I20260928 23:17:09.595813 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.593248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39121 (local address 127.0.98.3:49365)
0928 23:17:09.593299 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.593310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.593346 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.593456 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.593456 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.593457 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:17:09.593510 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.593512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.594047 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.594048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.594974 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.594977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.595456 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.595457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.595509 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.595522 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:17:09.595701 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:17:09.595702 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.595732 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.595764 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 23:17:09.595820 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.593292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49365 (local address 127.0.98.2:39121)
0928 23:17:09.593329 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.593331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.593339 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:09.593401 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.593402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.593411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.593453 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:09.593561 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.593989 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.595066 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.595380 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.595446 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.595620 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:09.595651 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:09.595699 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.595735 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.595753 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:17:09.605237 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:09.605342 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:09.606019 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066: Sending DeleteTablet for 3 replicas of tablet 6d2025fa116b4613a59492920a09c522
I20260928 23:17:09.606381 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d2025fa116b4613a59492920a09c522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49456
I20260928 23:17:09.606612 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d2025fa116b4613a59492920a09c522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48916
I20260928 23:17:09.606822 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d2025fa116b4613a59492920a09c522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58946
I20260928 23:17:09.607306 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:09.607442 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:09.607743 2665 tablet_replica.cc:330] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: stopping tablet replica
I20260928 23:17:09.607864 2665 raft_consensus.cc:2229] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:09.607918 2665 raft_consensus.cc:2258] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:09.608057 2666 tablet_replica.cc:330] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: stopping tablet replica
I20260928 23:17:09.608134 2666 raft_consensus.cc:2229] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:09.608217 2666 raft_consensus.cc:2258] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:09.608356 2667 tablet_replica.cc:330] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: stopping tablet replica
I20260928 23:17:09.608443 2667 raft_consensus.cc:2229] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:09.608485 2667 raft_consensus.cc:2258] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:09.608933 2394 tablet_service.cc:1467] Processing CreateTablet for tablet dde7975148dc4342947c67ff03ff6e6f (DEFAULT_TABLE table=test-table [id=884110c6892d45529b4c52a96b484c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:09.608974 2254 tablet_service.cc:1467] Processing CreateTablet for tablet dde7975148dc4342947c67ff03ff6e6f (DEFAULT_TABLE table=test-table [id=884110c6892d45529b4c52a96b484c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:09.609041 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde7975148dc4342947c67ff03ff6e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:09.609083 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde7975148dc4342947c67ff03ff6e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:09.610342 2534 tablet_service.cc:1467] Processing CreateTablet for tablet dde7975148dc4342947c67ff03ff6e6f (DEFAULT_TABLE table=test-table [id=884110c6892d45529b4c52a96b484c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:09.610366 2620 tablet_bootstrap.cc:492] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: Bootstrap starting.
I20260928 23:17:09.610443 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde7975148dc4342947c67ff03ff6e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:09.610740 2620 tablet_bootstrap.cc:654] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:09.611624 2622 tablet_bootstrap.cc:492] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: Bootstrap starting.
I20260928 23:17:09.611653 2621 tablet_bootstrap.cc:492] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9: Bootstrap starting.
I20260928 23:17:09.612028 2621 tablet_bootstrap.cc:654] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:09.612028 2622 tablet_bootstrap.cc:654] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:09.612484 2667 ts_tablet_manager.cc:1905] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:09.614033 2667 ts_tablet_manager.cc:1918] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:09.614091 2667 log.cc:1196] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/6d2025fa116b4613a59492920a09c522
I20260928 23:17:09.614318 2667 ts_tablet_manager.cc:1939] T 6d2025fa116b4613a59492920a09c522 P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting consensus metadata
I20260928 23:17:09.614703 2103 catalog_manager.cc:4841] TS 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3:44025): tablet 6d2025fa116b4613a59492920a09c522 (table test-table [id=20516b13204e4aa3b4b7b5245997cc1f]) successfully deleted
I20260928 23:17:09.615938 2621 tablet_bootstrap.cc:492] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9: No bootstrap required, opened a new log
I20260928 23:17:09.615981 2621 ts_tablet_manager.cc:1397] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 23:17:09.616108 2621 raft_consensus.cc:348] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.616169 2621 raft_consensus.cc:374] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:09.616187 2621 raft_consensus.cc:729] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf932d184934198b8bf59c3efbabfe9, State: Initialized, Role: FOLLOWER
I20260928 23:17:09.616226 2621 consensus_queue.cc:260] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.616313 2621 ts_tablet_manager.cc:1428] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.617385 2622 tablet_bootstrap.cc:492] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: No bootstrap required, opened a new log
I20260928 23:17:09.617426 2622 ts_tablet_manager.cc:1397] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260928 23:17:09.608955 2665 ts_tablet_manager.cc:1905] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:09.617550 2622 raft_consensus.cc:348] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.617605 2622 raft_consensus.cc:374] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:09.617622 2622 raft_consensus.cc:729] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f9a13feedf4d3ba37d0d672f5c38f1, State: Initialized, Role: FOLLOWER
I20260928 23:17:09.617658 2622 consensus_queue.cc:260] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.617730 2622 ts_tablet_manager.cc:1428] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:09.618992 2665 ts_tablet_manager.cc:1918] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:09.619062 2665 log.cc:1196] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/6d2025fa116b4613a59492920a09c522
I20260928 23:17:09.619258 2665 ts_tablet_manager.cc:1939] T 6d2025fa116b4613a59492920a09c522 P 3518f3b95b774723a329e691fa929aa5: Deleting consensus metadata
I20260928 23:17:09.619653 2101 catalog_manager.cc:4841] TS 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843): tablet 6d2025fa116b4613a59492920a09c522 (table test-table [id=20516b13204e4aa3b4b7b5245997cc1f]) successfully deleted
I20260928 23:17:09.619767 2666 ts_tablet_manager.cc:1905] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:09.621234 2666 ts_tablet_manager.cc:1918] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:09.621301 2666 log.cc:1196] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/6d2025fa116b4613a59492920a09c522
I20260928 23:17:09.621508 2666 ts_tablet_manager.cc:1939] T 6d2025fa116b4613a59492920a09c522 P 1bf932d184934198b8bf59c3efbabfe9: Deleting consensus metadata
I20260928 23:17:09.621863 2101 catalog_manager.cc:4841] TS 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121): tablet 6d2025fa116b4613a59492920a09c522 (table test-table [id=20516b13204e4aa3b4b7b5245997cc1f]) successfully deleted
I20260928 23:17:09.622118 2620 tablet_bootstrap.cc:492] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: No bootstrap required, opened a new log
I20260928 23:17:09.622161 2620 ts_tablet_manager.cc:1397] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260928 23:17:09.622287 2620 raft_consensus.cc:348] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.622344 2620 raft_consensus.cc:374] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:09.622361 2620 raft_consensus.cc:729] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3518f3b95b774723a329e691fa929aa5, State: Initialized, Role: FOLLOWER
I20260928 23:17:09.622402 2620 consensus_queue.cc:260] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.622476 2620 ts_tablet_manager.cc:1428] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260928 23:17:09.653424 2321 tablet.cc:2369] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:17:09.693883 2601 tablet.cc:2369] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:09.846894 2627 raft_consensus.cc:482] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:09.846980 2627 raft_consensus.cc:504] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.847092 2627 leader_election.cc:288] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843), 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121)
I20260928 23:17:09.847306 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde7975148dc4342947c67ff03ff6e6f" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518f3b95b774723a329e691fa929aa5" is_pre_election: true
I20260928 23:17:09.847318 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde7975148dc4342947c67ff03ff6e6f" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf932d184934198b8bf59c3efbabfe9" is_pre_election: true
I20260928 23:17:09.847419 2414 raft_consensus.cc:2454] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 0.
I20260928 23:17:09.847419 2274 raft_consensus.cc:2454] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 0.
I20260928 23:17:09.847575 2485 leader_election.cc:302] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf932d184934198b8bf59c3efbabfe9, 74f9a13feedf4d3ba37d0d672f5c38f1; no voters:
I20260928 23:17:09.847680 2627 raft_consensus.cc:2781] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:09.847741 2627 raft_consensus.cc:482] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:09.847761 2627 raft_consensus.cc:3037] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:09.848382 2627 raft_consensus.cc:504] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.848486 2627 leader_election.cc:288] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 election: Requested vote from peers 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843), 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121)
I20260928 23:17:09.848640 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde7975148dc4342947c67ff03ff6e6f" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518f3b95b774723a329e691fa929aa5"
I20260928 23:17:09.848657 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde7975148dc4342947c67ff03ff6e6f" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf932d184934198b8bf59c3efbabfe9"
I20260928 23:17:09.848723 2274 raft_consensus.cc:3037] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:09.848722 2414 raft_consensus.cc:3037] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:09.849452 2274 raft_consensus.cc:2454] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 1.
I20260928 23:17:09.849452 2414 raft_consensus.cc:2454] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 1.
I20260928 23:17:09.849615 2485 leader_election.cc:302] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf932d184934198b8bf59c3efbabfe9, 74f9a13feedf4d3ba37d0d672f5c38f1; no voters:
I20260928 23:17:09.849701 2627 raft_consensus.cc:2781] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:09.849749 2627 raft_consensus.cc:686] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 LEADER]: Becoming Leader. State: Replica: 74f9a13feedf4d3ba37d0d672f5c38f1, State: Running, Role: LEADER
I20260928 23:17:09.849798 2627 consensus_queue.cc:237] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:09.850302 2117 catalog_manager.cc:5495] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } health_report { overall_health: HEALTHY } } }
I20260928 23:17:09.904731 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.902655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58956 (local address 127.0.98.3:44025)
0928 23:17:09.902693 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.902695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.902719 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:17:09.902754 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.902755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.902763 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.902798 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:09.902885 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.903384 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.904069 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.904300 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.904365 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.904508 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:09.904540 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:09.904582 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.904632 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.904653 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:17:09.908496 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.906783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50055 (local address 127.0.98.3:44025)
0928 23:17:09.906828 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:09.906830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:09.906837 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:09.906852 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:09.906853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:09.906859 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:09.906895 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:09.906964 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.907369 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.907988 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.908168 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.908212 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.908314 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:09.908338 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:09.908371 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:09.908409 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:17:09.908426 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":25}
I20260928 23:17:09.908504 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:09.906725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44025 (local address 127.0.98.3:50055)
0928 23:17:09.906758 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:09.906769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:09.906809 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:09.906903 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:09.906904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:09.906905 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:17:09.906933 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.906933 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.907415 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.907415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.907933 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:09.907934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:09.908217 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:09.908218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:09.908249 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:09.908252 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:17:09.908378 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:17:09.908379 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 23:17:09.908407 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 23:17:09.908440 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":4}
I20260928 23:17:09.910116 2274 raft_consensus.cc:1261] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Refusing update from remote peer 74f9a13feedf4d3ba37d0d672f5c38f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:09.910118 2414 raft_consensus.cc:1261] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Refusing update from remote peer 74f9a13feedf4d3ba37d0d672f5c38f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:09.910320 2642 consensus_queue.cc:1035] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:09.910421 2627 consensus_queue.cc:1035] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:17:09.914072 2647 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 23:17:11.432183 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:11.432284 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:11.433074 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066: Sending DeleteTablet for 3 replicas of tablet dde7975148dc4342947c67ff03ff6e6f
I20260928 23:17:11.433347 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet dde7975148dc4342947c67ff03ff6e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58946
I20260928 23:17:11.433363 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet dde7975148dc4342947c67ff03ff6e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49456
I20260928 23:17:11.433523 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet dde7975148dc4342947c67ff03ff6e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48916
I20260928 23:17:11.433692 2698 tablet_replica.cc:330] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9: stopping tablet replica
I20260928 23:17:11.433749 2698 raft_consensus.cc:2229] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:11.433795 2698 raft_consensus.cc:2258] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:11.434007 2698 ts_tablet_manager.cc:1905] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:11.435487 2698 ts_tablet_manager.cc:1918] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:17:11.435642 2698 log.cc:1196] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/dde7975148dc4342947c67ff03ff6e6f
I20260928 23:17:11.435880 2698 ts_tablet_manager.cc:1939] T dde7975148dc4342947c67ff03ff6e6f P 1bf932d184934198b8bf59c3efbabfe9: Deleting consensus metadata
I20260928 23:17:11.435927 2696 tablet_replica.cc:330] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: stopping tablet replica
I20260928 23:17:11.435990 2696 raft_consensus.cc:2229] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:11.436075 2696 raft_consensus.cc:2258] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:11.436115 2101 catalog_manager.cc:4841] TS 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121): tablet dde7975148dc4342947c67ff03ff6e6f (table test-table [id=884110c6892d45529b4c52a96b484c13]) successfully deleted
I20260928 23:17:11.436316 2696 ts_tablet_manager.cc:1905] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:11.437423 2697 tablet_replica.cc:330] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: stopping tablet replica
I20260928 23:17:11.437490 2697 raft_consensus.cc:2229] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:11.437493 2696 ts_tablet_manager.cc:1918] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:17:11.437533 2696 log.cc:1196] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/dde7975148dc4342947c67ff03ff6e6f
I20260928 23:17:11.437567 2697 raft_consensus.cc:2258] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:11.437783 2696 ts_tablet_manager.cc:1939] T dde7975148dc4342947c67ff03ff6e6f P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting consensus metadata
I20260928 23:17:11.437789 2697 ts_tablet_manager.cc:1905] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:11.438079 2103 catalog_manager.cc:4841] TS 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3:44025): tablet dde7975148dc4342947c67ff03ff6e6f (table test-table [id=884110c6892d45529b4c52a96b484c13]) successfully deleted
I20260928 23:17:11.439122 2697 ts_tablet_manager.cc:1918] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:17:11.439177 2697 log.cc:1196] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/dde7975148dc4342947c67ff03ff6e6f
I20260928 23:17:11.439376 2697 ts_tablet_manager.cc:1939] T dde7975148dc4342947c67ff03ff6e6f P 3518f3b95b774723a329e691fa929aa5: Deleting consensus metadata
I20260928 23:17:11.439687 2101 catalog_manager.cc:4841] TS 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843): tablet dde7975148dc4342947c67ff03ff6e6f (table test-table [id=884110c6892d45529b4c52a96b484c13]) successfully deleted
Sep 28 23:17:11 dist-test-slave-djkr krb5kdc[2072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:17:11 dist-test-slave-djkr krb5kdc[2072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:11.466492 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:11.449196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.0.98.62:40993)
0928 23:17:11.449334 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:11.449336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:11.449345 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:11.449424 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:11.449425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:11.449436 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:11.449498 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:11.449999 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:11.450499 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:11.451194 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:11.451398 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:11.451451 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:11.464155 (+ 12704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:11.464158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:11.464158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:11.464167 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:11.465483 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:11.465788 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:11.465789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:11.465790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:11.465841 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:11.466005 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:11.466006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:11.466006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:11.466114 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:11.466200 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:11.466305 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 23:17:11.466388 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260928 23:17:11.467957 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:11.468113 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:11.469347 2394 tablet_service.cc:1467] Processing CreateTablet for tablet 42f9d7db247b4f89a688f1070dfd7242 (DEFAULT_TABLE table=test-table [id=9445d4bb30394767a4a95d4a2a488589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:11.469398 2254 tablet_service.cc:1467] Processing CreateTablet for tablet 42f9d7db247b4f89a688f1070dfd7242 (DEFAULT_TABLE table=test-table [id=9445d4bb30394767a4a95d4a2a488589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:11.469453 2534 tablet_service.cc:1467] Processing CreateTablet for tablet 42f9d7db247b4f89a688f1070dfd7242 (DEFAULT_TABLE table=test-table [id=9445d4bb30394767a4a95d4a2a488589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:11.469492 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f9d7db247b4f89a688f1070dfd7242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:11.469529 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f9d7db247b4f89a688f1070dfd7242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:11.469550 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f9d7db247b4f89a688f1070dfd7242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:11.470571 2707 tablet_bootstrap.cc:492] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5: Bootstrap starting.
I20260928 23:17:11.470912 2708 tablet_bootstrap.cc:492] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9: Bootstrap starting.
I20260928 23:17:11.470942 2707 tablet_bootstrap.cc:654] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:11.471002 2709 tablet_bootstrap.cc:492] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1: Bootstrap starting.
I20260928 23:17:11.471293 2708 tablet_bootstrap.cc:654] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:11.471314 2709 tablet_bootstrap.cc:654] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:11.471575 2707 tablet_bootstrap.cc:492] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5: No bootstrap required, opened a new log
I20260928 23:17:11.471629 2707 ts_tablet_manager.cc:1397] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:11.471761 2707 raft_consensus.cc:348] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.471837 2707 raft_consensus.cc:374] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:11.471850 2707 raft_consensus.cc:729] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3518f3b95b774723a329e691fa929aa5, State: Initialized, Role: FOLLOWER
I20260928 23:17:11.471895 2708 tablet_bootstrap.cc:492] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9: No bootstrap required, opened a new log
I20260928 23:17:11.471901 2707 consensus_queue.cc:260] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.471942 2708 ts_tablet_manager.cc:1397] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:11.472056 2707 ts_tablet_manager.cc:1428] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:11.472064 2708 raft_consensus.cc:348] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.472154 2708 raft_consensus.cc:374] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:11.472196 2708 raft_consensus.cc:729] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf932d184934198b8bf59c3efbabfe9, State: Initialized, Role: FOLLOWER
I20260928 23:17:11.472288 2708 consensus_queue.cc:260] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.472438 2709 tablet_bootstrap.cc:492] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1: No bootstrap required, opened a new log
I20260928 23:17:11.472466 2708 ts_tablet_manager.cc:1428] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:11.472517 2709 ts_tablet_manager.cc:1397] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:11.472651 2709 raft_consensus.cc:348] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.472772 2709 raft_consensus.cc:374] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:11.472909 2709 raft_consensus.cc:729] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f9a13feedf4d3ba37d0d672f5c38f1, State: Initialized, Role: FOLLOWER
I20260928 23:17:11.472993 2709 consensus_queue.cc:260] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.473129 2709 ts_tablet_manager.cc:1428] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:17:11.597468 2684 raft_consensus.cc:482] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:11.597559 2684 raft_consensus.cc:504] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.597708 2684 leader_election.cc:288] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843), 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121)
I20260928 23:17:11.597929 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f9d7db247b4f89a688f1070dfd7242" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf932d184934198b8bf59c3efbabfe9" is_pre_election: true
I20260928 23:17:11.597934 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f9d7db247b4f89a688f1070dfd7242" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518f3b95b774723a329e691fa929aa5" is_pre_election: true
I20260928 23:17:11.598021 2414 raft_consensus.cc:2454] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 0.
I20260928 23:17:11.598026 2274 raft_consensus.cc:2454] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 0.
I20260928 23:17:11.598162 2485 leader_election.cc:302] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf932d184934198b8bf59c3efbabfe9, 74f9a13feedf4d3ba37d0d672f5c38f1; no voters:
I20260928 23:17:11.598297 2684 raft_consensus.cc:2781] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:11.598343 2684 raft_consensus.cc:482] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:11.598357 2684 raft_consensus.cc:3037] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:11.599012 2684 raft_consensus.cc:504] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.599118 2684 leader_election.cc:288] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 election: Requested vote from peers 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843), 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121)
I20260928 23:17:11.599298 2414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f9d7db247b4f89a688f1070dfd7242" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf932d184934198b8bf59c3efbabfe9"
I20260928 23:17:11.599328 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f9d7db247b4f89a688f1070dfd7242" candidate_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518f3b95b774723a329e691fa929aa5"
I20260928 23:17:11.599375 2414 raft_consensus.cc:3037] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:11.599392 2274 raft_consensus.cc:3037] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:11.600083 2274 raft_consensus.cc:2454] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 1.
I20260928 23:17:11.600124 2414 raft_consensus.cc:2454] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f9a13feedf4d3ba37d0d672f5c38f1 in term 1.
I20260928 23:17:11.600237 2485 leader_election.cc:302] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3518f3b95b774723a329e691fa929aa5, 74f9a13feedf4d3ba37d0d672f5c38f1; no voters:
I20260928 23:17:11.600342 2684 raft_consensus.cc:2781] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:11.600392 2684 raft_consensus.cc:686] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 LEADER]: Becoming Leader. State: Replica: 74f9a13feedf4d3ba37d0d672f5c38f1, State: Running, Role: LEADER
I20260928 23:17:11.600437 2684 consensus_queue.cc:237] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.600991 2115 catalog_manager.cc:5495] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } health_report { overall_health: HEALTHY } } }
I20260928 23:17:11.618911 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:11.616747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.0.98.3:44025)
0928 23:17:11.616884 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:11.616886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:11.616893 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:11.616912 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:11.616912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:11.616920 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:11.616975 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:11.617108 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:11.617581 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:11.618287 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:11.618495 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:11.618547 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:11.618677 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:11.618710 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:11.618754 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:11.618811 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 23:17:11.618830 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260928 23:17:11.620568 2414 raft_consensus.cc:1261] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Refusing update from remote peer 74f9a13feedf4d3ba37d0d672f5c38f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:11.620584 2274 raft_consensus.cc:1261] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Refusing update from remote peer 74f9a13feedf4d3ba37d0d672f5c38f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:11.620740 2653 consensus_queue.cc:1035] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:11.620833 2719 consensus_queue.cc:1035] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:11.633172 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:11.633267 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:11.633828 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066: Sending DeleteTablet for 3 replicas of tablet 42f9d7db247b4f89a688f1070dfd7242
I20260928 23:17:11.634066 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f9d7db247b4f89a688f1070dfd7242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49456
I20260928 23:17:11.634123 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f9d7db247b4f89a688f1070dfd7242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48916
I20260928 23:17:11.634227 2697 tablet_replica.cc:330] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5: stopping tablet replica
I20260928 23:17:11.634289 2697 raft_consensus.cc:2229] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:11.634327 2697 raft_consensus.cc:2258] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:11.634269 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 23:17:11.634392 2698 tablet_replica.cc:330] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9: stopping tablet replica
W20260928 23:17:11.634400 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:11.634436 2698 raft_consensus.cc:2229] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:11.634471 2698 raft_consensus.cc:2258] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:11.634493 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f9d7db247b4f89a688f1070dfd7242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58946
I20260928 23:17:11.634563 2696 tablet_replica.cc:330] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1: stopping tablet replica
I20260928 23:17:11.634604 2696 raft_consensus.cc:2229] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:11.634646 2696 raft_consensus.cc:2258] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:11.634868 2696 ts_tablet_manager.cc:1905] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:11.635950 2697 ts_tablet_manager.cc:1905] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:11.636453 2696 ts_tablet_manager.cc:1918] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:11.636507 2696 log.cc:1196] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/42f9d7db247b4f89a688f1070dfd7242
I20260928 23:17:11.636507 2698 ts_tablet_manager.cc:1905] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:11.636700 2696 ts_tablet_manager.cc:1939] T 42f9d7db247b4f89a688f1070dfd7242 P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting consensus metadata
I20260928 23:17:11.636969 2103 catalog_manager.cc:4841] TS 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3:44025): tablet 42f9d7db247b4f89a688f1070dfd7242 (table test-table [id=9445d4bb30394767a4a95d4a2a488589]) successfully deleted
I20260928 23:17:11.637259 2697 ts_tablet_manager.cc:1918] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:11.637292 2697 log.cc:1196] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/42f9d7db247b4f89a688f1070dfd7242
I20260928 23:17:11.637449 2697 ts_tablet_manager.cc:1939] T 42f9d7db247b4f89a688f1070dfd7242 P 3518f3b95b774723a329e691fa929aa5: Deleting consensus metadata
I20260928 23:17:11.637841 2101 catalog_manager.cc:4841] TS 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843): tablet 42f9d7db247b4f89a688f1070dfd7242 (table test-table [id=9445d4bb30394767a4a95d4a2a488589]) successfully deleted
I20260928 23:17:11.638015 2698 ts_tablet_manager.cc:1918] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:11.638059 2698 log.cc:1196] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/42f9d7db247b4f89a688f1070dfd7242
I20260928 23:17:11.638212 2534 tablet_service.cc:1467] Processing CreateTablet for tablet b5a1378dcb03488b99da593ba51ad209 (DEFAULT_TABLE table=test-table [id=f5a2d2f61f4a4cc680f1dc552f86627f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:11.638240 2698 ts_tablet_manager.cc:1939] T 42f9d7db247b4f89a688f1070dfd7242 P 1bf932d184934198b8bf59c3efbabfe9: Deleting consensus metadata
I20260928 23:17:11.638291 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a1378dcb03488b99da593ba51ad209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:11.638463 2254 tablet_service.cc:1467] Processing CreateTablet for tablet b5a1378dcb03488b99da593ba51ad209 (DEFAULT_TABLE table=test-table [id=f5a2d2f61f4a4cc680f1dc552f86627f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:11.638546 2254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a1378dcb03488b99da593ba51ad209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:11.639315 2709 tablet_bootstrap.cc:492] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1: Bootstrap starting.
I20260928 23:17:11.639439 2394 tablet_service.cc:1467] Processing CreateTablet for tablet b5a1378dcb03488b99da593ba51ad209 (DEFAULT_TABLE table=test-table [id=f5a2d2f61f4a4cc680f1dc552f86627f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:11.639515 2394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a1378dcb03488b99da593ba51ad209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:11.639599 2707 tablet_bootstrap.cc:492] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5: Bootstrap starting.
I20260928 23:17:11.639632 2709 tablet_bootstrap.cc:654] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:11.639868 2707 tablet_bootstrap.cc:654] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:11.640147 2101 catalog_manager.cc:4841] TS 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121): tablet 42f9d7db247b4f89a688f1070dfd7242 (table test-table [id=9445d4bb30394767a4a95d4a2a488589]) successfully deleted
I20260928 23:17:11.640444 2707 tablet_bootstrap.cc:492] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5: No bootstrap required, opened a new log
I20260928 23:17:11.640444 2709 tablet_bootstrap.cc:492] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1: No bootstrap required, opened a new log
I20260928 23:17:11.640488 2709 ts_tablet_manager.cc:1397] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:17:11.640488 2707 ts_tablet_manager.cc:1397] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:11.640582 2709 raft_consensus.cc:348] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.640635 2709 raft_consensus.cc:374] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:11.640607 2707 raft_consensus.cc:348] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.640653 2709 raft_consensus.cc:729] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f9a13feedf4d3ba37d0d672f5c38f1, State: Initialized, Role: FOLLOWER
I20260928 23:17:11.640659 2707 raft_consensus.cc:374] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:11.640676 2707 raft_consensus.cc:729] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3518f3b95b774723a329e691fa929aa5, State: Initialized, Role: FOLLOWER
I20260928 23:17:11.640698 2709 consensus_queue.cc:260] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.640710 2707 consensus_queue.cc:260] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.640774 2709 ts_tablet_manager.cc:1428] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:11.640793 2707 ts_tablet_manager.cc:1428] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:11.640961 2708 tablet_bootstrap.cc:492] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9: Bootstrap starting.
I20260928 23:17:11.641273 2708 tablet_bootstrap.cc:654] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:11.641819 2708 tablet_bootstrap.cc:492] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9: No bootstrap required, opened a new log
I20260928 23:17:11.641870 2708 ts_tablet_manager.cc:1397] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:11.641994 2708 raft_consensus.cc:348] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.642055 2708 raft_consensus.cc:374] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:11.642071 2708 raft_consensus.cc:729] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf932d184934198b8bf59c3efbabfe9, State: Initialized, Role: FOLLOWER
I20260928 23:17:11.642107 2708 consensus_queue.cc:260] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.642189 2708 ts_tablet_manager.cc:1428] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:11.803833 2714 raft_consensus.cc:482] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:11.803928 2714 raft_consensus.cc:504] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.804085 2714 leader_election.cc:288] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843), 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3:44025)
I20260928 23:17:11.804277 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a1378dcb03488b99da593ba51ad209" candidate_uuid: "1bf932d184934198b8bf59c3efbabfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518f3b95b774723a329e691fa929aa5" is_pre_election: true
I20260928 23:17:11.804311 2551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a1378dcb03488b99da593ba51ad209" candidate_uuid: "1bf932d184934198b8bf59c3efbabfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" is_pre_election: true
I20260928 23:17:11.804368 2274 raft_consensus.cc:2454] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bf932d184934198b8bf59c3efbabfe9 in term 0.
I20260928 23:17:11.804400 2551 raft_consensus.cc:2454] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bf932d184934198b8bf59c3efbabfe9 in term 0.
I20260928 23:17:11.804517 2345 leader_election.cc:302] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf932d184934198b8bf59c3efbabfe9, 3518f3b95b774723a329e691fa929aa5; no voters:
I20260928 23:17:11.804620 2714 raft_consensus.cc:2781] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:11.804658 2714 raft_consensus.cc:482] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:11.804672 2714 raft_consensus.cc:3037] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:11.805334 2714 raft_consensus.cc:504] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.805451 2714 leader_election.cc:288] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [CANDIDATE]: Term 1 election: Requested vote from peers 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843), 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3:44025)
I20260928 23:17:11.805598 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a1378dcb03488b99da593ba51ad209" candidate_uuid: "1bf932d184934198b8bf59c3efbabfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3518f3b95b774723a329e691fa929aa5"
I20260928 23:17:11.805608 2551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a1378dcb03488b99da593ba51ad209" candidate_uuid: "1bf932d184934198b8bf59c3efbabfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1"
I20260928 23:17:11.805670 2551 raft_consensus.cc:3037] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:11.805670 2274 raft_consensus.cc:3037] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:11.806281 2274 raft_consensus.cc:2454] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf932d184934198b8bf59c3efbabfe9 in term 1.
I20260928 23:17:11.806285 2551 raft_consensus.cc:2454] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf932d184934198b8bf59c3efbabfe9 in term 1.
I20260928 23:17:11.806437 2345 leader_election.cc:302] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1bf932d184934198b8bf59c3efbabfe9, 3518f3b95b774723a329e691fa929aa5, 74f9a13feedf4d3ba37d0d672f5c38f1; no voters:
I20260928 23:17:11.806545 2714 raft_consensus.cc:2781] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:11.806639 2714 raft_consensus.cc:686] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 LEADER]: Becoming Leader. State: Replica: 1bf932d184934198b8bf59c3efbabfe9, State: Running, Role: LEADER
I20260928 23:17:11.806689 2714 consensus_queue.cc:237] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } }
I20260928 23:17:11.807273 2115 catalog_manager.cc:5495] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1bf932d184934198b8bf59c3efbabfe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bf932d184934198b8bf59c3efbabfe9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 } health_report { overall_health: UNKNOWN } } }
I20260928 23:17:11.824724 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:11.822602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.0.98.2:39121)
0928 23:17:11.822736 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:11.822738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:11.822745 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:11.822765 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:11.822766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:11.822775 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:11.822830 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:11.822923 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:11.823454 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:11.824179 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:11.824416 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:11.824468 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:11.824559 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:11.824584 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:11.824615 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:11.824651 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 23:17:11.824667 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260928 23:17:11.826661 2551 raft_consensus.cc:1261] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Refusing update from remote peer 1bf932d184934198b8bf59c3efbabfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:11.826661 2274 raft_consensus.cc:1261] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5 [term 1 FOLLOWER]: Refusing update from remote peer 1bf932d184934198b8bf59c3efbabfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:11.826877 2714 consensus_queue.cc:1035] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3518f3b95b774723a329e691fa929aa5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:11.826990 2725 consensus_queue.cc:1035] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74f9a13feedf4d3ba37d0d672f5c38f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:17:11.830304 2680 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 23:17:13.345837 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:13.345934 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:13.346660 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 856257ef9fa44ca0919c03eb45b98066: Sending DeleteTablet for 3 replicas of tablet b5a1378dcb03488b99da593ba51ad209
I20260928 23:17:13.346889 2254 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a1378dcb03488b99da593ba51ad209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49456
I20260928 23:17:13.346901 2394 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a1378dcb03488b99da593ba51ad209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48916
I20260928 23:17:13.346925 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a1378dcb03488b99da593ba51ad209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58946
I20260928 23:17:13.347030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 2189
I20260928 23:17:13.347069 2754 tablet_replica.cc:330] T b5a1378dcb03488b99da593ba51ad209 P 3518f3b95b774723a329e691fa929aa5: stopping tablet replica
I20260928 23:17:13.347610 2755 tablet_replica.cc:330] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9: stopping tablet replica
I20260928 23:17:13.347667 2755 raft_consensus.cc:2229] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:13.347723 2755 raft_consensus.cc:2258] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:13.347774 2756 tablet_replica.cc:330] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1: stopping tablet replica
I20260928 23:17:13.347829 2756 raft_consensus.cc:2229] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:13.347865 2756 raft_consensus.cc:2258] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:13.347987 2755 ts_tablet_manager.cc:1905] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:13.348069 2756 ts_tablet_manager.cc:1905] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:13.349568 2755 ts_tablet_manager.cc:1918] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:17:13.349625 2755 log.cc:1196] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/b5a1378dcb03488b99da593ba51ad209
I20260928 23:17:13.349617 2756 ts_tablet_manager.cc:1918] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:17:13.349661 2756 log.cc:1196] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/b5a1378dcb03488b99da593ba51ad209
I20260928 23:17:13.349848 2755 ts_tablet_manager.cc:1939] T b5a1378dcb03488b99da593ba51ad209 P 1bf932d184934198b8bf59c3efbabfe9: Deleting consensus metadata
I20260928 23:17:13.349890 2756 ts_tablet_manager.cc:1939] T b5a1378dcb03488b99da593ba51ad209 P 74f9a13feedf4d3ba37d0d672f5c38f1: Deleting consensus metadata
I20260928 23:17:13.350106 2101 catalog_manager.cc:4841] TS 1bf932d184934198b8bf59c3efbabfe9 (127.0.98.2:39121): tablet b5a1378dcb03488b99da593ba51ad209 (table test-table [id=f5a2d2f61f4a4cc680f1dc552f86627f]) successfully deleted
I20260928 23:17:13.350221 2103 catalog_manager.cc:4841] TS 74f9a13feedf4d3ba37d0d672f5c38f1 (127.0.98.3:44025): tablet b5a1378dcb03488b99da593ba51ad209 (table test-table [id=f5a2d2f61f4a4cc680f1dc552f86627f]) successfully deleted
W20260928 23:17:13.352912 2101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38843 (error 108)
I20260928 23:17:13.353055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 2329
I20260928 23:17:13.353787 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:13.353620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38843 (local address 127.0.0.1:44500)
0928 23:17:13.353715 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:13.353753 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":63,"threads_started":1}
W20260928 23:17:13.353895 2101 catalog_manager.cc:4584] TS 3518f3b95b774723a329e691fa929aa5 (127.0.98.1:38843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5a1378dcb03488b99da593ba51ad209: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38843: connect: Connection refused (error 111)
I20260928 23:17:13.358506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 2469
I20260928 23:17:13.363016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 2085
2026-09-28T23:17:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5537 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:13 dist-test-slave-djkr krb5kdc[2761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:13 dist-test-slave-djkr krb5kdc[2761](info): set up 2 sockets
Sep 28 23:17:13 dist-test-slave-djkr krb5kdc[2761](info): commencing operation
krb5kdc: starting...
W20260928 23:17:14.315033 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:14 dist-test-slave-djkr krb5kdc[2761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:14.421983 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44105
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:14.499234 2774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:14.499437 2774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:14.499459 2774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:14.500605 2774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:14.500644 2774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:14.500658 2774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:14.500669 2774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:14.500679 2774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:14.502161 2774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:14.502348 2774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:14.502521 2774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:14.507028 2780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:14.507046 2782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:14.507099 2779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:14.507347 2774 server_base.cc:1034] running on GCE node
I20260928 23:17:14.507601 2774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:14.507855 2774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:14.509009 2774 hybrid_clock.cc:648] HybridClock initialized: now 1790637434508993 us; error 30 us; skew 500 ppm
Sep 28 23:17:14 dist-test-slave-djkr krb5kdc[2761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:14.511284 2774 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:14.511736 2774 webserver.cc:460] Webserver started at http://127.0.98.62:37257/ using document root <none> and password file <none>
I20260928 23:17:14.511958 2774 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:14.512006 2774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:14.512131 2774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:14.513185 2774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "003d2f2072094b7f8257d8ba0e13e248"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "105bffed8e9174d48252b413c67a66d5"
server_key_iv: "effa86dc1b5ba119a7671c8a48361409"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.513510 2774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "003d2f2072094b7f8257d8ba0e13e248"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "105bffed8e9174d48252b413c67a66d5"
server_key_iv: "effa86dc1b5ba119a7671c8a48361409"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.514708 2774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:14.515341 2789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.515522 2774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:14.515627 2774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "003d2f2072094b7f8257d8ba0e13e248"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "105bffed8e9174d48252b413c67a66d5"
server_key_iv: "effa86dc1b5ba119a7671c8a48361409"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.515693 2774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:14.524825 2774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:14.526716 2774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:14.526852 2774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:14.530380 2774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44105
I20260928 23:17:14.530426 2841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44105 every 8 connection(s)
I20260928 23:17:14.530892 2774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:14.531535 2842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:14.533699 2842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248: Bootstrap starting.
I20260928 23:17:14.534204 2842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:14.534435 2842 log.cc:824] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:14.535086 2842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248: No bootstrap required, opened a new log
I20260928 23:17:14.536023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 2774
I20260928 23:17:14.536096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:14.536242 392 external_mini_cluster.cc:1428] Setting key 3a71d5c7a4bb5efea8789e39ec504cff
I20260928 23:17:14.537458 2842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003d2f2072094b7f8257d8ba0e13e248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } }
I20260928 23:17:14.537623 2842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:14.537699 2842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003d2f2072094b7f8257d8ba0e13e248, State: Initialized, Role: FOLLOWER
I20260928 23:17:14.537835 2842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003d2f2072094b7f8257d8ba0e13e248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } }
I20260928 23:17:14.537909 2842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:14.537930 2842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:14.537973 2842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:14.538563 2842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003d2f2072094b7f8257d8ba0e13e248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } }
I20260928 23:17:14.538664 2842 leader_election.cc:302] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 003d2f2072094b7f8257d8ba0e13e248; no voters:
I20260928 23:17:14.538825 2842 leader_election.cc:288] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:14.538878 2847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:14.539021 2847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [term 1 LEADER]: Becoming Leader. State: Replica: 003d2f2072094b7f8257d8ba0e13e248, State: Running, Role: LEADER
I20260928 23:17:14.539083 2842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:14.539110 2847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003d2f2072094b7f8257d8ba0e13e248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } }
I20260928 23:17:14.539422 2847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "003d2f2072094b7f8257d8ba0e13e248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003d2f2072094b7f8257d8ba0e13e248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } } }
I20260928 23:17:14.539474 2848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 003d2f2072094b7f8257d8ba0e13e248. Latest consensus state: current_term: 1 leader_uuid: "003d2f2072094b7f8257d8ba0e13e248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003d2f2072094b7f8257d8ba0e13e248" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44105 } } }
I20260928 23:17:14.539620 2848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:14.539530 2847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:14.539819 2855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:14.541074 2855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:14.542366 2855 catalog_manager.cc:1347] Generated new cluster ID: 03dbe1399c0a4e758cc29220298ce393
I20260928 23:17:14.542412 2855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:17:14 dist-test-slave-djkr krb5kdc[2761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:14.547576 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:14.536727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.0.98.62:44105)
0928 23:17:14.536939 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:14.536950 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:14.536987 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:17:14.537430 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:14.537439 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:14.537502 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:14.537648 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:14.538313 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.539747 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.540529 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.541130 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.541231 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:14.544041 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:14.544045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:14.544046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:14.544057 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:14.545919 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:14.546262 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:14.546265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:14.546265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:14.546383 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:14.546551 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:14.546552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:14.546552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:14.546879 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:14.547041 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:14.547108 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 23:17:14.547343 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":127,"threads_started":1}
I20260928 23:17:14.551137 2855 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:14.551877 2855 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:14.560346 2855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 003d2f2072094b7f8257d8ba0e13e248: Generated new TSK 0
I20260928 23:17:14.560539 2855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:17:14.588593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44105
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:14.665467 2868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:14.665663 2868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:14.665683 2868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:14.666797 2868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:14.666838 2868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:14.666869 2868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:14.668336 2868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:14.668541 2868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:14.668749 2868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:14.673432 2876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:14.673456 2873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:14.673447 2874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:14.673659 2868 server_base.cc:1034] running on GCE node
I20260928 23:17:14.673954 2868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:14.674170 2868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:14.675333 2868 hybrid_clock.cc:648] HybridClock initialized: now 1790637434675291 us; error 34 us; skew 500 ppm
Sep 28 23:17:14 dist-test-slave-djkr krb5kdc[2761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:14.677742 2868 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:17:14.678149 2868 webserver.cc:460] Webserver started at http://127.0.98.1:33153/ using document root <none> and password file <none>
I20260928 23:17:14.678326 2868 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:14.678365 2868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:14.678455 2868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:14.679438 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "a255ae0e6ae34ad6ad0849d547fe0439"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "679d3e54e8023aa59583ca0bb339e85f"
server_key_iv: "186999d48e50f31e0267a990ffad1d38"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.679787 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a255ae0e6ae34ad6ad0849d547fe0439"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "679d3e54e8023aa59583ca0bb339e85f"
server_key_iv: "186999d48e50f31e0267a990ffad1d38"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.680944 2868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:14.681591 2883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.681736 2868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.681805 2868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "a255ae0e6ae34ad6ad0849d547fe0439"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "679d3e54e8023aa59583ca0bb339e85f"
server_key_iv: "186999d48e50f31e0267a990ffad1d38"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.681872 2868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:14.692867 2868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:14.694485 2868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:14.694599 2868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:14.694783 2868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:14.695044 2868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:14.695075 2868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.695093 2868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:14.695104 2868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.700254 2868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39077
I20260928 23:17:14.700330 2996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39077 every 8 connection(s)
I20260928 23:17:14.700577 2868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:14.702548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 2868
I20260928 23:17:14.702613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:14.702759 392 external_mini_cluster.cc:1428] Setting key 4db7147ec228108fbfa9e0219913c275
Sep 28 23:17:14 dist-test-slave-djkr krb5kdc[2761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:14.709666 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:14.701190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39471 (local address 127.0.98.62:44105)
0928 23:17:14.701271 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:14.701273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:14.701411 (+ 138us) server_negotiation.cc:407] Connection header received
0928 23:17:14.702317 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:14.702319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:14.702330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:14.702376 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:14.703069 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.703698 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.704824 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.705031 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.705080 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:14.707139 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:14.707141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:14.707141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:14.707148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:14.708471 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:14.708830 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:14.708831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:14.708832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:14.708880 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:14.709080 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:14.709081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:14.709081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:14.709186 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:14.709239 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:14.709493 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 23:17:14.709577 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:17:14.709817 2999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:14.701149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44105 (local address 127.0.98.1:39471)
0928 23:17:14.701284 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:14.701328 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:14.702214 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:14.702454 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:14.702458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:14.702749 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:14.702998 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:14.703004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.703775 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.703776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:14.704706 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:14.704709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.705098 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.705099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:14.705245 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:14.705665 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:14.705674 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:14.707053 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:14.708541 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:14.708548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:14.708550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:14.708750 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:14.708930 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:14.708931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:14.708932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:14.709028 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:14.709237 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:14.709243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:14.709378 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 23:17:14.709463 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:17:14.709615 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:17:14.710304 2997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44105
I20260928 23:17:14.710402 2997 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:14.710597 2997 heartbeater.cc:511] Master 127.0.98.62:44105 requested a full tablet report, sending...
I20260928 23:17:14.711059 2806 ts_manager.cc:194] Registered new tserver with Master: a255ae0e6ae34ad6ad0849d547fe0439 (127.0.98.1:39077)
I20260928 23:17:14.711761 2806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:17:14.726708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44105
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:14.805516 3002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:14.805706 3002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:14.805725 3002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:14.806795 3002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:14.806828 3002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:14.806851 3002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:14.808322 3002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:14.808504 3002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:14.808669 3002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:14.813159 3008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:14.813202 3007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:14.813313 3002 server_base.cc:1034] running on GCE node
W20260928 23:17:14.813177 3010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:14.813522 3002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:14.813759 3002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:14.814919 3002 hybrid_clock.cc:648] HybridClock initialized: now 1790637434814900 us; error 32 us; skew 500 ppm
Sep 28 23:17:14 dist-test-slave-djkr krb5kdc[2761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:14.817370 3002 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:17:14.817806 3002 webserver.cc:460] Webserver started at http://127.0.98.2:34477/ using document root <none> and password file <none>
I20260928 23:17:14.818009 3002 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:14.818058 3002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:14.818179 3002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:14.819247 3002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "82430565442347fc9e5cf699461bff98"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "091696162985279ea3357df29628fd07"
server_key_iv: "5d75eb4e08a8ecc1a949203dbc80d3c9"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.819644 3002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "82430565442347fc9e5cf699461bff98"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "091696162985279ea3357df29628fd07"
server_key_iv: "5d75eb4e08a8ecc1a949203dbc80d3c9"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.820837 3002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:14.821506 3017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.821650 3002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:14.821727 3002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "82430565442347fc9e5cf699461bff98"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "091696162985279ea3357df29628fd07"
server_key_iv: "5d75eb4e08a8ecc1a949203dbc80d3c9"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.821798 3002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:14.839291 3002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:14.841540 3002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:14.841682 3002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:14.841895 3002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:14.842170 3002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:14.842201 3002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.842231 3002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:14.842250 3002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.847514 3002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39325
I20260928 23:17:14.847582 3130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39325 every 8 connection(s)
I20260928 23:17:14.847883 3002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:14.850929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 3002
I20260928 23:17:14.851004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:14.851177 392 external_mini_cluster.cc:1428] Setting key 233cbc3c03af0db4891f57d8bc02d72d
Sep 28 23:17:14 dist-test-slave-djkr krb5kdc[2761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:14.857312 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:14.848520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54921 (local address 127.0.98.62:44105)
0928 23:17:14.848578 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:14.848580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:14.848779 (+ 199us) server_negotiation.cc:407] Connection header received
0928 23:17:14.849699 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:14.849701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:14.849713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:14.849765 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:14.850395 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.850879 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.852070 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.852260 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.852315 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:14.854571 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:14.854572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:14.854573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:14.854579 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:14.855831 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:14.856301 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:14.856301 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:14.856302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:14.856347 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:14.856636 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:14.856637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:14.856637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:14.856754 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:14.856818 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:14.857164 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 23:17:14.857247 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:17:14.857510 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:14.848512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44105 (local address 127.0.98.2:54921)
0928 23:17:14.848658 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:14.848694 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:14.849588 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:14.849836 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:14.849840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:14.850092 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:14.850273 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:14.850277 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.850962 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.850963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:14.851998 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:14.852001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.852325 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.852326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:14.852451 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:14.852943 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:14.852953 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:14.854474 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:14.855902 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:14.855909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:14.855911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:14.856205 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:14.856405 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:14.856406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:14.856406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:14.856542 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:14.856827 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:14.856832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:14.857003 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 23:17:14.857118 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 23:17:14.857284 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260928 23:17:14.858011 3131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44105
I20260928 23:17:14.858103 3131 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:14.858270 3131 heartbeater.cc:511] Master 127.0.98.62:44105 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:17:14.858619 2806 ts_manager.cc:194] Registered new tserver with Master: 82430565442347fc9e5cf699461bff98 (127.0.98.2:39325)
I20260928 23:17:14.858960 2806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:17:14.874179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44105
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:14.951495 3136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:14.951740 3136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:14.951762 3136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:14.952884 3136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:14.952917 3136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:14.952941 3136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:14.954376 3136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:14.954555 3136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:14.954727 3136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:14.959440 3144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:14.959455 3142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:14.959437 3141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:14.959600 3136 server_base.cc:1034] running on GCE node
I20260928 23:17:14.959882 3136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:14.960088 3136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:14.961227 3136 hybrid_clock.cc:648] HybridClock initialized: now 1790637434961214 us; error 28 us; skew 500 ppm
Sep 28 23:17:14 dist-test-slave-djkr krb5kdc[2761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:14.963495 3136 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:17:14.963927 3136 webserver.cc:460] Webserver started at http://127.0.98.3:41335/ using document root <none> and password file <none>
I20260928 23:17:14.964102 3136 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:14.964138 3136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:14.964226 3136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:14.965262 3136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "c8da30021aac4c3d97de9a56cd301c97"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "74fee52d7103ea83f67b33eb6b3ae63f"
server_key_iv: "77c8556543fa390f2e35491bbd8098be"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.965548 3136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c8da30021aac4c3d97de9a56cd301c97"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "74fee52d7103ea83f67b33eb6b3ae63f"
server_key_iv: "77c8556543fa390f2e35491bbd8098be"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.966648 3136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:14.967237 3151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.967401 3136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.967481 3136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "c8da30021aac4c3d97de9a56cd301c97"
format_stamp: "Formatted at 2026-09-28 23:17:14 on dist-test-slave-djkr"
server_key: "74fee52d7103ea83f67b33eb6b3ae63f"
server_key_iv: "77c8556543fa390f2e35491bbd8098be"
server_key_version: "encryptionkey@0"
I20260928 23:17:14.967562 3136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:14.975661 3136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:14.977605 3136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:14.977718 3136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:14.977916 3136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:14.978184 3136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:14.978217 3136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.978248 3136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:14.978273 3136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:14.983405 3136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39945
I20260928 23:17:14.983464 3264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39945 every 8 connection(s)
I20260928 23:17:14.983757 3136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:14.988217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 3136
I20260928 23:17:14.988322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:14.988492 392 external_mini_cluster.cc:1428] Setting key 5ed4cf075b29c0a9dc5119c14110cc15
Sep 28 23:17:14 dist-test-slave-djkr krb5kdc[2761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:14.992107 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:14.984431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36657 (local address 127.0.98.62:44105)
0928 23:17:14.984486 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:14.984488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:14.984683 (+ 195us) server_negotiation.cc:407] Connection header received
0928 23:17:14.985585 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:14.985586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:14.985596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:14.985646 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:14.986188 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.986699 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.987625 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.987798 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.987846 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:14.989935 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:14.989936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:14.989936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:14.989942 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:14.990933 (+ 991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:14.991236 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:14.991237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:14.991237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:14.991279 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:14.991504 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:14.991505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:14.991505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:14.991638 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:14.991720 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:14.991984 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 23:17:14.992044 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:17:14.992272 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:14.984358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44105 (local address 127.0.98.3:36657)
0928 23:17:14.984545 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:14.984607 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:14.985498 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:14.985707 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:14.985710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:14.985945 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:14.986122 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:14.986126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.986760 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.986761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:14.987532 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:14.987533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:14.987853 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:14.987854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:14.987950 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:14.988433 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:14.988440 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:14.989859 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:14.990990 (+ 1131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:14.990994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:14.990995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:14.991183 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:14.991320 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:14.991320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:14.991321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:14.991454 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:14.991733 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:14.991739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:14.991862 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 23:17:14.991958 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:17:14.992075 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260928 23:17:14.992772 3265 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44105
I20260928 23:17:14.992898 3265 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:14.993063 3265 heartbeater.cc:511] Master 127.0.98.62:44105 requested a full tablet report, sending...
I20260928 23:17:14.993404 2806 ts_manager.cc:194] Registered new tserver with Master: c8da30021aac4c3d97de9a56cd301c97 (127.0.98.3:39945)
I20260928 23:17:14.993728 2806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36657
I20260928 23:17:15.000638 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:17:15.010390 2846 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:15.009623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.0.98.62:44105)
0928 23:17:15.009673 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:15.009675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:15.009759 (+ 84us) server_negotiation.cc:407] Connection header received
0928 23:17:15.009812 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:15.009813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:15.009823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:15.009891 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:15.010322 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43486: BlockingRecv error: recv got EOF from 127.0.0.1:43486 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:17:15.011293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 2868
I20260928 23:17:15.015094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 3002
I20260928 23:17:15.018719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 3136
I20260928 23:17:15.022076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 2774
2026-09-28T23:17:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1658 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:15 dist-test-slave-djkr krb5kdc[3277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:15 dist-test-slave-djkr krb5kdc[3277](info): set up 2 sockets
Sep 28 23:17:15 dist-test-slave-djkr krb5kdc[3277](info): commencing operation
krb5kdc: starting...
W20260928 23:17:15.967150 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:16 dist-test-slave-djkr krb5kdc[3277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:16.076025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45479
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:16.152956 3290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:16.153151 3290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:16.153178 3290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:16.154306 3290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:16.154345 3290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:16.154361 3290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:16.154376 3290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:16.154390 3290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:16.155858 3290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:16.156095 3290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:16.156293 3290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:16.160693 3298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:16.160708 3295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:16.160795 3296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:16.160953 3290 server_base.cc:1034] running on GCE node
I20260928 23:17:16.161176 3290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:16.161432 3290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:16.162572 3290 hybrid_clock.cc:648] HybridClock initialized: now 1790637436162553 us; error 32 us; skew 500 ppm
Sep 28 23:17:16 dist-test-slave-djkr krb5kdc[3277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:16.164878 3290 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:16.165272 3290 webserver.cc:460] Webserver started at http://127.0.98.62:45545/ using document root <none> and password file <none>
I20260928 23:17:16.165460 3290 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:16.165508 3290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:16.165623 3290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:16.166591 3290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "1d317e39cf4a4fdaa01ad5cb74055b70"
format_stamp: "Formatted at 2026-09-28 23:17:16 on dist-test-slave-djkr"
server_key: "3a457e96ccf1c300ff8054cb117cf437"
server_key_iv: "9041c715c90e63bf1b9308b34220aeb5"
server_key_version: "encryptionkey@0"
I20260928 23:17:16.166908 3290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d317e39cf4a4fdaa01ad5cb74055b70"
format_stamp: "Formatted at 2026-09-28 23:17:16 on dist-test-slave-djkr"
server_key: "3a457e96ccf1c300ff8054cb117cf437"
server_key_iv: "9041c715c90e63bf1b9308b34220aeb5"
server_key_version: "encryptionkey@0"
I20260928 23:17:16.168210 3290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:16.168881 3305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:16.169035 3290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:16.169121 3290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "1d317e39cf4a4fdaa01ad5cb74055b70"
format_stamp: "Formatted at 2026-09-28 23:17:16 on dist-test-slave-djkr"
server_key: "3a457e96ccf1c300ff8054cb117cf437"
server_key_iv: "9041c715c90e63bf1b9308b34220aeb5"
server_key_version: "encryptionkey@0"
I20260928 23:17:16.169180 3290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:16.180573 3290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:16.182466 3290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:16.182595 3290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:16.186084 3290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45479
I20260928 23:17:16.186124 3357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45479 every 8 connection(s)
I20260928 23:17:16.186419 3290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:16.187199 3358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:16.189312 3358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70: Bootstrap starting.
I20260928 23:17:16.189723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 3290
I20260928 23:17:16.189805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:16.189910 3358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:16.189965 392 external_mini_cluster.cc:1428] Setting key 106f54bce6dbe92ad5aa7ee13b56de1d
I20260928 23:17:16.190164 3358 log.cc:824] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:16.190790 3358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70: No bootstrap required, opened a new log
I20260928 23:17:16.192739 3358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d317e39cf4a4fdaa01ad5cb74055b70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45479 } }
I20260928 23:17:16.192909 3358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:16.192953 3358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d317e39cf4a4fdaa01ad5cb74055b70, State: Initialized, Role: FOLLOWER
I20260928 23:17:16.193073 3358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d317e39cf4a4fdaa01ad5cb74055b70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45479 } }
I20260928 23:17:16.193149 3358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:16.193186 3358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:16.193228 3358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:16.193832 3358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d317e39cf4a4fdaa01ad5cb74055b70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45479 } }
I20260928 23:17:16.193935 3358 leader_election.cc:302] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d317e39cf4a4fdaa01ad5cb74055b70; no voters:
I20260928 23:17:16.194080 3358 leader_election.cc:288] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:16.194113 3363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:16.194221 3363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [term 1 LEADER]: Becoming Leader. State: Replica: 1d317e39cf4a4fdaa01ad5cb74055b70, State: Running, Role: LEADER
I20260928 23:17:16.194293 3363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d317e39cf4a4fdaa01ad5cb74055b70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45479 } }
I20260928 23:17:16.194316 3358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:16.194576 3363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d317e39cf4a4fdaa01ad5cb74055b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d317e39cf4a4fdaa01ad5cb74055b70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45479 } } }
I20260928 23:17:16.194644 3363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:16.195482 3374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 23:17:16 dist-test-slave-djkr krb5kdc[3277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:16.194598 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d317e39cf4a4fdaa01ad5cb74055b70. Latest consensus state: current_term: 1 leader_uuid: "1d317e39cf4a4fdaa01ad5cb74055b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d317e39cf4a4fdaa01ad5cb74055b70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45479 } } }
I20260928 23:17:16.196643 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:16.195941 3374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:16.198458 3374 catalog_manager.cc:1347] Generated new cluster ID: cd181ba6e72646fea7f45f5c6b844ab1
I20260928 23:17:16.198529 3374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:16.199887 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:16.190373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.0.98.62:45479)
0928 23:17:16.190525 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:16.190534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:16.190568 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:17:16.190986 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:16.190992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:16.191038 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:16.191150 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:16.191874 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:16.192715 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:16.193720 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:16.194174 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:16.194277 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:16.196634 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:16.196639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:16.196639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:16.196651 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:16.198307 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:16.198683 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:16.198686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:16.198686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:16.198799 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:16.198989 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:16.198990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:16.198990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:16.199290 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:16.199454 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:16.199525 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 23:17:16.199683 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260928 23:17:16.200709 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:17:16.205991 3374 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:16.206663 3374 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:16.213416 3374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d317e39cf4a4fdaa01ad5cb74055b70: Generated new TSK 0
I20260928 23:17:16.213572 3374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:17:16.219836 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:16.216358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.0.98.62:45479)
0928 23:17:16.216409 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:16.216411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:16.216489 (+ 78us) server_negotiation.cc:407] Connection header received
0928 23:17:16.216560 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:16.216561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:16.216568 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:16.216618 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:16.217015 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:16.217514 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:16.218579 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:16.218788 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:16.218835 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:16.219432 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:16.219433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:16.219434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:16.219628 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:16.219644 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:16.219649 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:16.219694 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:16.219762 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:17:16.219785 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:17:16.222340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 3290
2026-09-28T23:17:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1198 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:16 dist-test-slave-djkr krb5kdc[3391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:16 dist-test-slave-djkr krb5kdc[3391](info): set up 2 sockets
Sep 28 23:17:16 dist-test-slave-djkr krb5kdc[3391](info): commencing operation
krb5kdc: starting...
W20260928 23:17:17.167155 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:17.269948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:17.345588 3404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:17.345770 3404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:17.345798 3404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:17.346879 3404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:17.346916 3404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:17.346935 3404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:17.346949 3404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:17.346966 3404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:17.348393 3404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:17.348613 3404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:17.348793 3404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:17.353104 3409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:17.353127 3410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:17.353111 3412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:17.353483 3404 server_base.cc:1034] running on GCE node
I20260928 23:17:17.353701 3404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:17.353964 3404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:17.355100 3404 hybrid_clock.cc:648] HybridClock initialized: now 1790637437355084 us; error 29 us; skew 500 ppm
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:17.363152 3404 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:17.363602 3404 webserver.cc:460] Webserver started at http://127.0.98.62:43571/ using document root <none> and password file <none>
I20260928 23:17:17.363802 3404 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:17.363857 3404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:17.363979 3404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:17.364946 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "d7a3d723a10346698ec08a4672cb6290"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "659d1e91ce7dd6a57c6eab594a6f9f13"
server_key_iv: "717e9934a21b6b5f1bdba0cf5575934d"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.365269 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7a3d723a10346698ec08a4672cb6290"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "659d1e91ce7dd6a57c6eab594a6f9f13"
server_key_iv: "717e9934a21b6b5f1bdba0cf5575934d"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.366472 3404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:17.367089 3419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.367218 3404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.367293 3404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "d7a3d723a10346698ec08a4672cb6290"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "659d1e91ce7dd6a57c6eab594a6f9f13"
server_key_iv: "717e9934a21b6b5f1bdba0cf5575934d"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.367358 3404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:17.377790 3404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:17.379314 3404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:17.379441 3404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:17.382965 3404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40447
I20260928 23:17:17.382994 3471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40447 every 8 connection(s)
I20260928 23:17:17.383279 3404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:17.383507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 3404
I20260928 23:17:17.383622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:17.383798 392 external_mini_cluster.cc:1428] Setting key 4fb734bbe457fc8f564481736045b539
I20260928 23:17:17.384362 3472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:17.387001 3472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290: Bootstrap starting.
I20260928 23:17:17.387655 3472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:17.387918 3472 log.cc:824] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:17.388521 3472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290: No bootstrap required, opened a new log
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:17.390421 3472 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a3d723a10346698ec08a4672cb6290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40447 } }
I20260928 23:17:17.390547 3472 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:17.390568 3472 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a3d723a10346698ec08a4672cb6290, State: Initialized, Role: FOLLOWER
I20260928 23:17:17.390676 3472 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a3d723a10346698ec08a4672cb6290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40447 } }
I20260928 23:17:17.390746 3472 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:17.390776 3472 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:17.390822 3472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:17.391283 3472 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a3d723a10346698ec08a4672cb6290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40447 } }
I20260928 23:17:17.391376 3472 leader_election.cc:302] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7a3d723a10346698ec08a4672cb6290; no voters:
I20260928 23:17:17.391522 3472 leader_election.cc:288] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:17.391583 3477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:17.391705 3477 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [term 1 LEADER]: Becoming Leader. State: Replica: d7a3d723a10346698ec08a4672cb6290, State: Running, Role: LEADER
I20260928 23:17:17.391817 3472 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:17.391836 3477 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a3d723a10346698ec08a4672cb6290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40447 } }
I20260928 23:17:17.392215 3479 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7a3d723a10346698ec08a4672cb6290. Latest consensus state: current_term: 1 leader_uuid: "d7a3d723a10346698ec08a4672cb6290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a3d723a10346698ec08a4672cb6290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40447 } } }
I20260928 23:17:17.392360 3479 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:17.392390 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7a3d723a10346698ec08a4672cb6290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a3d723a10346698ec08a4672cb6290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40447 } } }
I20260928 23:17:17.392684 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:17.392911 3482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:17.393343 3482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:17.393848 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:17.384261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.0.98.62:40447)
0928 23:17:17.384411 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:17.384419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:17.384450 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:17:17.384866 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:17.384872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:17.384914 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:17.385016 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:17.385723 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.386565 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.387582 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.388024 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.388117 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:17.390276 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:17.390280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:17.390280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:17.390291 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:17.392040 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:17.392462 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:17.392464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:17.392465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:17.392564 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:17.392957 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:17.392958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:17.392958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:17.393276 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:17.393453 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:17.393538 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 23:17:17.393672 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260928 23:17:17.394724 3482 catalog_manager.cc:1347] Generated new cluster ID: 4b4d07caffc242c8b9cd8b0342a91597
I20260928 23:17:17.394762 3482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:17.406895 3482 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:17.407351 3482 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:17.413702 3482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7a3d723a10346698ec08a4672cb6290: Generated new TSK 0
I20260928 23:17:17.413844 3482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:17:17.440410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40447
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:17.523051 3498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:17.523252 3498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:17.523273 3498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:17.524464 3498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:17.524504 3498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:17.524533 3498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:17.525943 3498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:17.526162 3498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:17.526368 3498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:17.530992 3503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:17.530999 3506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:17.531010 3504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:17.531026 3498 server_base.cc:1034] running on GCE node
I20260928 23:17:17.531469 3498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:17.531701 3498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:17.532858 3498 hybrid_clock.cc:648] HybridClock initialized: now 1790637437532846 us; error 30 us; skew 500 ppm
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:17.535339 3498 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:17:17.535777 3498 webserver.cc:460] Webserver started at http://127.0.98.1:32781/ using document root <none> and password file <none>
I20260928 23:17:17.535948 3498 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:17.535984 3498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:17.536072 3498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:17.537035 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "c9d27ba7d40d4a4b84fb1d7bf49598d3"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "9e3b88bc5996cef9e0e47656d1eaba70"
server_key_iv: "ceb72c8652beab69ccb85c741a5d1892"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.537320 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c9d27ba7d40d4a4b84fb1d7bf49598d3"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "9e3b88bc5996cef9e0e47656d1eaba70"
server_key_iv: "ceb72c8652beab69ccb85c741a5d1892"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.538491 3498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:17.539146 3513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.539343 3498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:17.539438 3498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "c9d27ba7d40d4a4b84fb1d7bf49598d3"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "9e3b88bc5996cef9e0e47656d1eaba70"
server_key_iv: "ceb72c8652beab69ccb85c741a5d1892"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.539497 3498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:17.552788 3498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:17.554360 3498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:17.554483 3498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:17.554672 3498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:17.554940 3498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:17.554972 3498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.554998 3498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:17.555015 3498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.560185 3498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39671
I20260928 23:17:17.560241 3626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39671 every 8 connection(s)
I20260928 23:17:17.560518 3498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:17.564723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 3498
I20260928 23:17:17.564801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:17.564980 392 external_mini_cluster.cc:1428] Setting key b411a29673bce4d3cace5c7cfbc0905a
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:17.568940 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:17.561144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46159 (local address 127.0.98.62:40447)
0928 23:17:17.561199 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:17.561201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:17.561359 (+ 158us) server_negotiation.cc:407] Connection header received
0928 23:17:17.561814 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:17.561815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:17.561824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:17.561878 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:17.562721 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.563207 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.564158 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.564356 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.564407 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:17.566521 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:17.566522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:17.566523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:17.566530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:17.567776 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:17.568110 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:17.568111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:17.568111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:17.568159 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:17.568368 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:17.568369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:17.568369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:17.568470 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:17.568524 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:17.568776 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 23:17:17.568854 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:17:17.569093 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:17.561094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40447 (local address 127.0.98.1:46159)
0928 23:17:17.561239 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:17.561280 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:17.561740 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:17.561939 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:17.561943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:17.562475 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:17.562654 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:17.562659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.563263 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.563264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:17.564098 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:17.564100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.564423 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.564424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:17.564524 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:17.565023 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:17.565032 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:17.566436 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:17.567843 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:17.567850 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:17.567852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:17.568049 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:17.568211 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:17.568212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:17.568212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:17.568315 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:17.568524 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:17.568530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:17.568658 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 23:17:17.568742 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:17:17.568889 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260928 23:17:17.569525 3627 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40447
I20260928 23:17:17.569638 3627 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:17.569816 3627 heartbeater.cc:511] Master 127.0.98.62:40447 requested a full tablet report, sending...
I20260928 23:17:17.570367 3436 ts_manager.cc:194] Registered new tserver with Master: c9d27ba7d40d4a4b84fb1d7bf49598d3 (127.0.98.1:39671)
I20260928 23:17:17.570961 3436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46159
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:17:17.586278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40447
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:17.664863 3632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:17.665066 3632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:17.665094 3632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:17.666234 3632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:17.666275 3632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:17.666311 3632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:17.667742 3632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:17.667977 3632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:17.668177 3632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:17.672979 3637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:17.672995 3638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:17.673002 3640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:17.673302 3632 server_base.cc:1034] running on GCE node
I20260928 23:17:17.673477 3632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:17.673691 3632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:17.674832 3632 hybrid_clock.cc:648] HybridClock initialized: now 1790637437674813 us; error 25 us; skew 500 ppm
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:17.677217 3632 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:17:17.677621 3632 webserver.cc:460] Webserver started at http://127.0.98.2:33441/ using document root <none> and password file <none>
I20260928 23:17:17.677820 3632 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:17.677866 3632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:17.677987 3632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:17.678972 3632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "f636f93fdbb94101aaae28369babccf6"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "4526c29d41a6277d977b724498bc61d3"
server_key_iv: "dd9474a9e4025356c32cb6c627c062e0"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.679296 3632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f636f93fdbb94101aaae28369babccf6"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "4526c29d41a6277d977b724498bc61d3"
server_key_iv: "dd9474a9e4025356c32cb6c627c062e0"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.680583 3632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:17:17.681273 3647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.681432 3632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:17.681509 3632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "f636f93fdbb94101aaae28369babccf6"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "4526c29d41a6277d977b724498bc61d3"
server_key_iv: "dd9474a9e4025356c32cb6c627c062e0"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.681574 3632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:17.690743 3632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:17.692879 3632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:17.693011 3632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:17.693240 3632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:17.693519 3632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:17.693549 3632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.693570 3632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:17.693588 3632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.698865 3632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35859
I20260928 23:17:17.698901 3760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35859 every 8 connection(s)
I20260928 23:17:17.699182 3632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:17.700892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 3632
I20260928 23:17:17.700997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:17.701180 392 external_mini_cluster.cc:1428] Setting key 6f0ce8b76b8c0d57bd51586eb2964bf9
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:17:17.708155 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:17.699840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35869 (local address 127.0.98.62:40447)
0928 23:17:17.699879 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:17.699881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:17.700077 (+ 196us) server_negotiation.cc:407] Connection header received
0928 23:17:17.700540 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:17.700541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:17.700552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:17.700601 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:17.701681 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.702194 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.703443 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.703698 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.703749 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:17.705717 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:17.705719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:17.705719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:17.705726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:17.706951 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:17.707293 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:17.707294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:17.707294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:17.707350 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:17.707576 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:17.707577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:17.707578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:17.707687 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:17.707740 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:17.707990 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 23:17:17.708066 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:17:17.708328 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:17.699832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40447 (local address 127.0.98.2:35869)
0928 23:17:17.699957 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:17.699991 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:17.700458 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:17.700655 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:17.700658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:17.701363 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:17.701602 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:17.701609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.702285 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.702286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:17.703366 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:17.703368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.703773 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.703774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:17.703889 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:17.704351 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:17.704359 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:17.705636 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:17.707020 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:17.707026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:17.707028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:17.707230 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:17.707400 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:17.707401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:17.707401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:17.707503 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:17.707737 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:17.707743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:17.707875 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 23:17:17.707963 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:17:17.708110 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20260928 23:17:17.708745 3761 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40447
I20260928 23:17:17.708829 3761 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:17.708989 3761 heartbeater.cc:511] Master 127.0.98.62:40447 requested a full tablet report, sending...
I20260928 23:17:17.709347 3436 ts_manager.cc:194] Registered new tserver with Master: f636f93fdbb94101aaae28369babccf6 (127.0.98.2:35859)
I20260928 23:17:17.709674 3436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:17:17.723697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40447
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:17.800947 3766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:17.801137 3766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:17.801165 3766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:17.802274 3766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:17.802312 3766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:17.802345 3766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:17.803717 3766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:17.803936 3766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:17.804127 3766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:17:17.808704 3766 server_base.cc:1034] running on GCE node
W20260928 23:17:17.808671 3771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:17.808686 3774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:17.808665 3772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:17.809096 3766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:17.809338 3766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:17.810482 3766 hybrid_clock.cc:648] HybridClock initialized: now 1790637437810470 us; error 30 us; skew 500 ppm
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:17.812821 3766 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:17:17.813194 3766 webserver.cc:460] Webserver started at http://127.0.98.3:35647/ using document root <none> and password file <none>
I20260928 23:17:17.813383 3766 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:17.813428 3766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:17.813545 3766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:17.814495 3766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "72f6b86376444501830c4bc8c1f0936c"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "33fc8b8e57a256e23eb186df153532e7"
server_key_iv: "8978e34c2408ef7ac7b9f3b5c4119f4b"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.814810 3766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "72f6b86376444501830c4bc8c1f0936c"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "33fc8b8e57a256e23eb186df153532e7"
server_key_iv: "8978e34c2408ef7ac7b9f3b5c4119f4b"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.816017 3766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:17:17.816620 3781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.816771 3766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:17.816865 3766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "72f6b86376444501830c4bc8c1f0936c"
format_stamp: "Formatted at 2026-09-28 23:17:17 on dist-test-slave-djkr"
server_key: "33fc8b8e57a256e23eb186df153532e7"
server_key_iv: "8978e34c2408ef7ac7b9f3b5c4119f4b"
server_key_version: "encryptionkey@0"
I20260928 23:17:17.816926 3766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:17.834862 3766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:17.836711 3766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:17.836843 3766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:17.837033 3766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:17.837312 3766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:17.837349 3766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.837378 3766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:17.837401 3766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:17.842584 3766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46139
I20260928 23:17:17.842638 3894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46139 every 8 connection(s)
I20260928 23:17:17.842926 3766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:17.847954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 3766
I20260928 23:17:17.848027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:17.848178 392 external_mini_cluster.cc:1428] Setting key 19d6a1a47d887cc8149bacf53f1f18cd
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:17.850970 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:17.843592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50821 (local address 127.0.98.62:40447)
0928 23:17:17.843640 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:17.843642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:17.843815 (+ 173us) server_negotiation.cc:407] Connection header received
0928 23:17:17.844263 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:17.844264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:17.844275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:17.844322 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:17.845150 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.845639 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.846571 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.846769 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.846820 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:17.848797 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:17.848798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:17.848799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:17.848807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:17.849818 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:17.850152 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:17.850153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:17.850154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:17.850210 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:17.850410 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:17.850411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:17.850411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:17.850509 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:17.850563 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:17.850811 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 23:17:17.850884 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:17:17.851143 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:17.843580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40447 (local address 127.0.98.3:50821)
0928 23:17:17.843710 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:17.843742 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:17.844189 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:17.844372 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:17.844375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:17.844905 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:17.845087 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:17.845091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.845701 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.845702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:17.846499 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:17.846501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.846834 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.846834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:17.846925 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:17.847311 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:17.847318 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:17.848729 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:17.849885 (+ 1156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:17.849891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:17.849894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:17.850091 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:17.850262 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:17.850262 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:17.850262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:17.850359 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:17.850559 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:17.850565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:17.850689 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 23:17:17.850782 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 23:17:17.850921 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260928 23:17:17.851531 3895 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40447
I20260928 23:17:17.851649 3895 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:17.851807 3895 heartbeater.cc:511] Master 127.0.98.62:40447 requested a full tablet report, sending...
I20260928 23:17:17.852109 3436 ts_manager.cc:194] Registered new tserver with Master: 72f6b86376444501830c4bc8c1f0936c (127.0.98.3:46139)
I20260928 23:17:17.852478 3436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50821
I20260928 23:17:17.859800 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637437, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:17.880064 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:17.873665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.0.98.62:40447)
0928 23:17:17.873722 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:17.873724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:17.873840 (+ 116us) server_negotiation.cc:407] Connection header received
0928 23:17:17.873867 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:17.873868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:17.873878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:17.873929 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:17.874423 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.874901 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.875655 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:17.875841 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:17.875898 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:17.877958 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:17.877959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:17.877959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:17.877965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:17.879029 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:17.879354 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:17.879355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:17.879355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:17.879405 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:17.879634 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:17.879635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:17.879635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:17.879782 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:17.879859 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:17.879941 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 23:17:17.879997 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 23:17:17.880291 3436 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57818
I20260928 23:17:17.880631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 3498
I20260928 23:17:17.884272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 3632
I20260928 23:17:17.887902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 3766
I20260928 23:17:17.891409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 3404
2026-09-28T23:17:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1670 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3907](info): set up 2 sockets
Sep 28 23:17:17 dist-test-slave-djkr krb5kdc[3907](info): commencing operation
krb5kdc: starting...
W20260928 23:17:18.831130 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:18 dist-test-slave-djkr krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:18.940629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44755
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:19.018467 3920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:19.018653 3920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:19.018673 3920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:19.019822 3920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:19.019860 3920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:19.019871 3920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:19.019882 3920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:19.019893 3920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:19.021271 3920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:19.021476 3920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:19.021674 3920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:19.026257 3926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:19.026230 3928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:19.026275 3925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:19.026302 3920 server_base.cc:1034] running on GCE node
I20260928 23:17:19.026623 3920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:19.026909 3920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:19.028076 3920 hybrid_clock.cc:648] HybridClock initialized: now 1790637439028062 us; error 28 us; skew 500 ppm
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:19.030413 3920 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:19.030819 3920 webserver.cc:460] Webserver started at http://127.0.98.62:39407/ using document root <none> and password file <none>
I20260928 23:17:19.031024 3920 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:19.031072 3920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:19.031190 3920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:19.032253 3920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "f9b0948ffe98487fb494600d87615153"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "733a9211f79c763389932a21c5080fb0"
server_key_iv: "d2db98de7f89f4e16e2e9ed4778eb538"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.032575 3920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9b0948ffe98487fb494600d87615153"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "733a9211f79c763389932a21c5080fb0"
server_key_iv: "d2db98de7f89f4e16e2e9ed4778eb538"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.033762 3920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:19.034397 3935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.034554 3920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:19.034631 3920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "f9b0948ffe98487fb494600d87615153"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "733a9211f79c763389932a21c5080fb0"
server_key_iv: "d2db98de7f89f4e16e2e9ed4778eb538"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.034709 3920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:19.056639 3920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:19.058406 3920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:19.058535 3920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:19.062217 3920 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44755
I20260928 23:17:19.062248 3987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44755 every 8 connection(s)
I20260928 23:17:19.062585 3920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:19.063416 3988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:19.064685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 3920
I20260928 23:17:19.064772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:19.064932 392 external_mini_cluster.cc:1428] Setting key 5910b83bddb65c19a3b9000bef22259a
I20260928 23:17:19.065852 3988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153: Bootstrap starting.
I20260928 23:17:19.066378 3988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:19.066627 3988 log.cc:824] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:19.067219 3988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153: No bootstrap required, opened a new log
I20260928 23:17:19.068923 3988 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b0948ffe98487fb494600d87615153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44755 } }
I20260928 23:17:19.069051 3988 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:19.069082 3988 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b0948ffe98487fb494600d87615153, State: Initialized, Role: FOLLOWER
I20260928 23:17:19.069218 3988 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b0948ffe98487fb494600d87615153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44755 } }
I20260928 23:17:19.069281 3988 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:19.069309 3988 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:19.069355 3988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:19.069823 3988 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b0948ffe98487fb494600d87615153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44755 } }
I20260928 23:17:19.069921 3988 leader_election.cc:302] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b0948ffe98487fb494600d87615153; no voters:
I20260928 23:17:19.070061 3988 leader_election.cc:288] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:19.070091 3993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:19.070194 3993 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [term 1 LEADER]: Becoming Leader. State: Replica: f9b0948ffe98487fb494600d87615153, State: Running, Role: LEADER
I20260928 23:17:19.070312 3988 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:19.070327 3993 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b0948ffe98487fb494600d87615153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44755 } }
I20260928 23:17:19.070700 3993 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b0948ffe98487fb494600d87615153. Latest consensus state: current_term: 1 leader_uuid: "f9b0948ffe98487fb494600d87615153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b0948ffe98487fb494600d87615153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44755 } } }
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:19.070950 3993 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:19.070884 3995 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b0948ffe98487fb494600d87615153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b0948ffe98487fb494600d87615153" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44755 } } }
I20260928 23:17:19.071219 3995 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:19.071386 4002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:19.071920 4002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:19.073704 4002 catalog_manager.cc:1347] Generated new cluster ID: 4847a5e4bcfc4e90b386b9fc89761a99
I20260928 23:17:19.073745 4002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:19.074887 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:19.065398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.0.98.62:44755)
0928 23:17:19.065571 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:19.065579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:19.065602 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:17:19.066004 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:19.066010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:19.066058 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:19.066182 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:19.066797 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.067744 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.068445 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.068825 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.068915 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:19.071489 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:19.071494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:19.071494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:19.071507 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:19.073348 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.073673 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.073676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.073677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.073777 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.073998 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.074000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.074000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.074296 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:19.074431 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:19.074558 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 23:17:19.074678 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260928 23:17:19.080370 4002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:19.081034 4002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:19.084740 4002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b0948ffe98487fb494600d87615153: Generated new TSK 0
I20260928 23:17:19.084879 4002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:17:19.116214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44755
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:19.193764 4014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:19.193959 4014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:19.193988 4014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:19.195125 4014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:19.195163 4014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:19.195197 4014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:19.196638 4014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:19.196864 4014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:19.197057 4014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:17:19.201647 4014 server_base.cc:1034] running on GCE node
W20260928 23:17:19.201751 4019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:19.201731 4020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:19.201683 4022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:19.202006 4014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:19.202214 4014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:19.203372 4014 hybrid_clock.cc:648] HybridClock initialized: now 1790637439203362 us; error 29 us; skew 500 ppm
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:19.205832 4014 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:17:19.206228 4014 webserver.cc:460] Webserver started at http://127.0.98.1:33121/ using document root <none> and password file <none>
I20260928 23:17:19.206405 4014 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:19.206446 4014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:19.206552 4014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:19.207490 4014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "233517affa9e49af8ed22d7078617ab1"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "b8379efef53a0faa4419ae0249339056"
server_key_iv: "9ae6f833b9b2f699fe231b7f4e91c29f"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.207844 4014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "233517affa9e49af8ed22d7078617ab1"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "b8379efef53a0faa4419ae0249339056"
server_key_iv: "9ae6f833b9b2f699fe231b7f4e91c29f"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.209020 4014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:19.209651 4029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.209811 4014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:19.209883 4014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "233517affa9e49af8ed22d7078617ab1"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "b8379efef53a0faa4419ae0249339056"
server_key_iv: "9ae6f833b9b2f699fe231b7f4e91c29f"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.209945 4014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:19.220088 4014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:19.221937 4014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:19.222046 4014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:19.222244 4014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:19.222496 4014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:19.222527 4014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.222546 4014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:19.222569 4014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.227924 4014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40843
I20260928 23:17:19.228039 4142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40843 every 8 connection(s)
I20260928 23:17:19.228253 4014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:19.230551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 4014
I20260928 23:17:19.230635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:19.230808 392 external_mini_cluster.cc:1428] Setting key 921db4d4df1025806e3384286319ba7c
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:19.237080 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:19.228824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55689 (local address 127.0.98.62:44755)
0928 23:17:19.228873 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:19.228876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:19.229034 (+ 158us) server_negotiation.cc:407] Connection header received
0928 23:17:19.229955 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:19.229957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:19.229969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:19.230018 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:19.230592 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.231200 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.232154 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.232390 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.232450 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:19.234472 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:19.234473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:19.234473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:19.234480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:19.235852 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.236197 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.236198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.236199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.236265 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.236459 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.236460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.236460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.236585 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:19.236645 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:19.236902 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 23:17:19.236988 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:17:19.237143 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:19.228794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44755 (local address 127.0.98.1:55689)
0928 23:17:19.228923 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:19.228959 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:19.229848 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:19.230085 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:19.230089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:19.230320 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:19.230506 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:19.230510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.231276 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.231277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:19.232075 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:19.232077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.232493 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.232494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:19.232613 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:19.233018 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:19.233026 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:19.234388 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:19.235932 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:19.235937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:19.235938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:19.236135 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:19.236321 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:19.236322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:19.236322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:19.236407 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:19.236644 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:19.236647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:19.236789 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 23:17:19.236867 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:17:19.236993 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
I20260928 23:17:19.237701 4143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44755
I20260928 23:17:19.237783 4143 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:19.237939 4143 heartbeater.cc:511] Master 127.0.98.62:44755 requested a full tablet report, sending...
I20260928 23:17:19.238485 3952 ts_manager.cc:194] Registered new tserver with Master: 233517affa9e49af8ed22d7078617ab1 (127.0.98.1:40843)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:17:19.239233 3952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:17:19.252357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44755
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:19.328559 4148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:19.328740 4148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:19.328760 4148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:19.329844 4148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:19.329881 4148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:19.329906 4148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:19.331287 4148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:19.331480 4148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:19.331682 4148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:19.336175 4154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:19.336221 4153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:19.336226 4148 server_base.cc:1034] running on GCE node
W20260928 23:17:19.336167 4156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:19.336501 4148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:19.336683 4148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:19.337809 4148 hybrid_clock.cc:648] HybridClock initialized: now 1790637439337794 us; error 29 us; skew 500 ppm
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:19.340034 4148 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:17:19.340399 4148 webserver.cc:460] Webserver started at http://127.0.98.2:33795/ using document root <none> and password file <none>
I20260928 23:17:19.340554 4148 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:19.340590 4148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:19.340673 4148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:19.341564 4148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "381c5c24057344f88e1c26a4304bfbba"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "a41270a8c3d4cc06a46b1416a228658d"
server_key_iv: "f95c405ea0b01cd2e7e5041f3fd64f2b"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.341837 4148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "381c5c24057344f88e1c26a4304bfbba"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "a41270a8c3d4cc06a46b1416a228658d"
server_key_iv: "f95c405ea0b01cd2e7e5041f3fd64f2b"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.342878 4148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:19.343423 4163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.343591 4148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.343665 4148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "381c5c24057344f88e1c26a4304bfbba"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "a41270a8c3d4cc06a46b1416a228658d"
server_key_iv: "f95c405ea0b01cd2e7e5041f3fd64f2b"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.343714 4148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:19.351840 4148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:19.353281 4148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:19.353384 4148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:19.353531 4148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:19.353744 4148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:19.353771 4148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.353789 4148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:19.353806 4148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.358815 4148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46491
I20260928 23:17:19.358876 4276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46491 every 8 connection(s)
I20260928 23:17:19.359100 4148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:19.366074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 4148
I20260928 23:17:19.366163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:19.366338 392 external_mini_cluster.cc:1428] Setting key 8e385a82e9fee62c8e413e3c88024fa7
I20260928 23:17:19.366894 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:19.359733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50201 (local address 127.0.98.62:44755)
0928 23:17:19.359776 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:19.359778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:19.359978 (+ 200us) server_negotiation.cc:407] Connection header received
0928 23:17:19.360773 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:19.360773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:19.360782 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:19.360833 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:19.361334 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.361813 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.362668 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.362841 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.362879 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:19.364632 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:19.364633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:19.364633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:19.364639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:19.365652 (+ 1013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.365997 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.365998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.365998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.366058 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.366291 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.366292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.366292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.366417 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:19.366483 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:19.366739 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 23:17:19.366817 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:17:19.367026 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:19.359715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44755 (local address 127.0.98.2:50201)
0928 23:17:19.359876 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:19.359907 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:19.360697 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:19.360890 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:19.360893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:19.361111 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:19.361275 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:19.361279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.361868 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.361868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:19.362612 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:19.362614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.362892 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.362892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:19.362977 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:19.363326 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:19.363333 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:19.364557 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:19.365717 (+ 1160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:19.365722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:19.365723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:19.365940 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:19.366114 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:19.366115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:19.366115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:19.366232 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:19.366473 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:19.366478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:19.366623 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 23:17:19.366707 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:17:19.366814 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260928 23:17:19.367412 4277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44755
I20260928 23:17:19.367501 4277 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:19.367681 4277 heartbeater.cc:511] Master 127.0.98.62:44755 requested a full tablet report, sending...
I20260928 23:17:19.368009 3952 ts_manager.cc:194] Registered new tserver with Master: 381c5c24057344f88e1c26a4304bfbba (127.0.98.2:46491)
I20260928 23:17:19.368428 3952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50201
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:17:19.387953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44755
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:19.463860 4282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:19.464061 4282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:19.464084 4282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:19.465265 4282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:19.465322 4282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:19.465358 4282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:19.466791 4282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:19.467003 4282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:19.467202 4282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:19.471875 4290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:19.471882 4287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:19.471926 4282 server_base.cc:1034] running on GCE node
W20260928 23:17:19.471913 4288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:19.472216 4282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:19.472406 4282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:19.473568 4282 hybrid_clock.cc:648] HybridClock initialized: now 1790637439473558 us; error 25 us; skew 500 ppm
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:19.475893 4282 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:17:19.476265 4282 webserver.cc:460] Webserver started at http://127.0.98.3:39469/ using document root <none> and password file <none>
I20260928 23:17:19.476460 4282 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:19.476502 4282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:19.476615 4282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:19.477586 4282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "4f799741bb894beda8439c60f3f03949"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "b289dd54b1349ecf57580a2c93921266"
server_key_iv: "bfc4fcbfbf734812eb481af1ebd8511b"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.477893 4282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4f799741bb894beda8439c60f3f03949"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "b289dd54b1349ecf57580a2c93921266"
server_key_iv: "bfc4fcbfbf734812eb481af1ebd8511b"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.479076 4282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:19.479681 4297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.479809 4282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:19.479895 4282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "4f799741bb894beda8439c60f3f03949"
format_stamp: "Formatted at 2026-09-28 23:17:19 on dist-test-slave-djkr"
server_key: "b289dd54b1349ecf57580a2c93921266"
server_key_iv: "bfc4fcbfbf734812eb481af1ebd8511b"
server_key_version: "encryptionkey@0"
I20260928 23:17:19.479959 4282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:19.500916 4282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:19.503039 4282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:19.503173 4282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:19.503376 4282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:19.503696 4282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:19.503731 4282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.503762 4282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:19.503782 4282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:19.508941 4282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37727
I20260928 23:17:19.509001 4410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37727 every 8 connection(s)
I20260928 23:17:19.509274 4282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:19.511565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 4282
I20260928 23:17:19.511646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:19.511824 392 external_mini_cluster.cc:1428] Setting key 98a3f77e9b1eb4e57d722006b9b8384c
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:19.517639 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:19.509924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55943 (local address 127.0.98.62:44755)
0928 23:17:19.509974 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:19.509977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:19.510166 (+ 189us) server_negotiation.cc:407] Connection header received
0928 23:17:19.511039 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:19.511040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:19.511050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:19.511098 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:19.511736 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.512403 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.513413 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.513613 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.513690 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:19.515630 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:19.515631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:19.515631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:19.515638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:19.516664 (+ 1026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.516985 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.516986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.516986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.517028 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.517193 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.517193 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.517193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.517280 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:19.517323 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:19.517525 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 23:17:19.517577 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:17:19.517802 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:19.509904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44755 (local address 127.0.98.3:55943)
0928 23:17:19.510052 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:19.510089 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:19.510958 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:19.511154 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:19.511158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:19.511396 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:19.511625 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:19.511631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.512476 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.512477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:19.513332 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:19.513334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.513697 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.513698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:19.513824 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:19.514267 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:19.514275 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:19.515525 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:19.516725 (+ 1200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:19.516730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:19.516731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:19.516932 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:19.517070 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:19.517070 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:19.517071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:19.517153 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:19.517327 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:19.517329 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:19.517436 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 23:17:19.517510 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:17:19.517617 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 23:17:19.518184 4411 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44755
I20260928 23:17:19.518270 4411 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:19.518429 4411 heartbeater.cc:511] Master 127.0.98.62:44755 requested a full tablet report, sending...
I20260928 23:17:19.518729 3952 ts_manager.cc:194] Registered new tserver with Master: 4f799741bb894beda8439c60f3f03949 (127.0.98.3:37727)
I20260928 23:17:19.519065 3952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55943
I20260928 23:17:19.523247 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:17:19.543792 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:19.537052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.0.98.1:40843)
0928 23:17:19.537199 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:19.537204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:19.537215 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:17:19.537241 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:19.537243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:19.537261 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:19.537340 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:19.537808 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.538563 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.539317 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.539659 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.539711 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:19.541657 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:19.541664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:19.541664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:19.541670 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:19.542892 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.543174 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.543176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.543176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.543223 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.543348 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.543348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.543348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.543458 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:19.543578 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:19.543639 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 23:17:19.543699 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:19.550483 3991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:19.544880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.0.98.62:44755)
0928 23:17:19.544920 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:19.544922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:19.544931 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:19.544977 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:19.544978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:19.544985 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:19.545029 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:19.545425 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.545909 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.546571 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:19.546754 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:19.546806 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:19.548627 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:19.548628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:19.548628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:19.548635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:19.549670 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.549934 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.549934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.549935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.549976 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:19.550089 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:19.550090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:19.550090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:19.550208 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:19.550272 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:19.550351 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 23:17:19.550408 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 23:17:19.550701 3952 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34954
I20260928 23:17:19.551002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 4014
I20260928 23:17:19.555364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 4148
I20260928 23:17:19.558797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 4282
I20260928 23:17:19.562270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 3920
2026-09-28T23:17:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1669 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[4423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[4423](info): set up 2 sockets
Sep 28 23:17:19 dist-test-slave-djkr krb5kdc[4423](info): commencing operation
krb5kdc: starting...
W20260928 23:17:20.511240 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:20 dist-test-slave-djkr krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:20.616677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44595
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:20.692571 4436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:20.692772 4436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:20.692801 4436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:20.693924 4436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:20.693962 4436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:20.693979 4436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:20.693995 4436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:20.694010 4436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:20.695430 4436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:20.695681 4436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:20.695876 4436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:20.700201 4441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:20.700207 4444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:20.700227 4442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:20.700249 4436 server_base.cc:1034] running on GCE node
I20260928 23:17:20.700618 4436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:20.700857 4436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:20.701999 4436 hybrid_clock.cc:648] HybridClock initialized: now 1790637440701985 us; error 27 us; skew 500 ppm
Sep 28 23:17:20 dist-test-slave-djkr krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:20.704176 4436 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:20.704444 4436 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:20.704485 4436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:20.704572 4436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:20.705554 4436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "7cbd3f1f502542848d9da675e5382a13"
format_stamp: "Formatted at 2026-09-28 23:17:20 on dist-test-slave-djkr"
server_key: "8b350065c08ca41ab9fcbccbf7708060"
server_key_iv: "29e6f64cbce760f07dba4fc8ac42dd1c"
server_key_version: "encryptionkey@0"
I20260928 23:17:20.705832 4436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "7cbd3f1f502542848d9da675e5382a13"
format_stamp: "Formatted at 2026-09-28 23:17:20 on dist-test-slave-djkr"
server_key: "8b350065c08ca41ab9fcbccbf7708060"
server_key_iv: "29e6f64cbce760f07dba4fc8ac42dd1c"
server_key_version: "encryptionkey@0"
I20260928 23:17:20.706914 4436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:20.707521 4450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:20.707682 4436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:20.707760 4436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "7cbd3f1f502542848d9da675e5382a13"
format_stamp: "Formatted at 2026-09-28 23:17:20 on dist-test-slave-djkr"
server_key: "8b350065c08ca41ab9fcbccbf7708060"
server_key_iv: "29e6f64cbce760f07dba4fc8ac42dd1c"
server_key_version: "encryptionkey@0"
I20260928 23:17:20.707820 4436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:20.717796 4436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:20.719497 4436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:20.719669 4436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:20.723054 4436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44595
I20260928 23:17:20.723081 4502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44595 every 8 connection(s)
I20260928 23:17:20.723315 4436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:20.724159 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:20.726327 4503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13: Bootstrap starting.
I20260928 23:17:20.726879 4503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:20.727129 4503 log.cc:824] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:20.727747 4503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13: No bootstrap required, opened a new log
I20260928 23:17:20.729215 4503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbd3f1f502542848d9da675e5382a13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } }
I20260928 23:17:20.729347 4503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:20.729377 4503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cbd3f1f502542848d9da675e5382a13, State: Initialized, Role: FOLLOWER
I20260928 23:17:20.729511 4503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbd3f1f502542848d9da675e5382a13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } }
I20260928 23:17:20.729564 4503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:20.729584 4503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:20.729628 4503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:20.730368 4503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbd3f1f502542848d9da675e5382a13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } }
I20260928 23:17:20.730438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 4436
I20260928 23:17:20.730473 4503 leader_election.cc:302] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cbd3f1f502542848d9da675e5382a13; no voters:
I20260928 23:17:20.730530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:20.730626 4503 leader_election.cc:288] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:20.730664 4506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:20.730715 392 external_mini_cluster.cc:1428] Setting key a11f2a4feaa68e3093d696e1dd5aaa4a
I20260928 23:17:20.730820 4506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [term 1 LEADER]: Becoming Leader. State: Replica: 7cbd3f1f502542848d9da675e5382a13, State: Running, Role: LEADER
I20260928 23:17:20.730870 4503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:20.730916 4506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbd3f1f502542848d9da675e5382a13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } }
I20260928 23:17:20.731299 4508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cbd3f1f502542848d9da675e5382a13. Latest consensus state: current_term: 1 leader_uuid: "7cbd3f1f502542848d9da675e5382a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbd3f1f502542848d9da675e5382a13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } } }
I20260928 23:17:20.731374 4508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:20.731388 4507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cbd3f1f502542848d9da675e5382a13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cbd3f1f502542848d9da675e5382a13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } } }
I20260928 23:17:20.731446 4507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:20.731676 4512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:20.732332 4512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:20.733826 4512 catalog_manager.cc:1347] Generated new cluster ID: 8ffb40f30f734981bb8aa3a1c39348d5
I20260928 23:17:20.733887 4512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:17:20 dist-test-slave-djkr krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:20.740139 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:20.731126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.0.98.62:44595)
0928 23:17:20.731275 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:20.731285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:20.731330 (+ 45us) server_negotiation.cc:407] Connection header received
0928 23:17:20.731443 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:20.731449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:20.731503 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:20.732222 (+ 719us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:20.732424 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:20.733686 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:20.734495 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:20.734927 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:20.735020 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:20.737051 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:20.737055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:20.737055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:20.737067 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:20.738557 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:20.738964 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:20.738967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:20.738967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:20.739065 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:20.739255 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:20.739256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:20.739256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:20.739525 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:20.739719 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:20.739794 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 23:17:20.740001 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260928 23:17:20.746120 4512 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:20.746587 4512 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:20.748718 4512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13: Generated new TSK 0
I20260928 23:17:20.748880 4512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:17:20.773969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:20.852421 4529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:20.852612 4529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:20.852643 4529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:20.853777 4529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:20.853817 4529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:20.853852 4529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:20.855312 4529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:20.855528 4529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:20.855777 4529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:20.860267 4535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:20.860272 4534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:20.860467 4529 server_base.cc:1034] running on GCE node
W20260928 23:17:20.860272 4537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:20.860709 4529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:20.860924 4529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:20.862059 4529 hybrid_clock.cc:648] HybridClock initialized: now 1790637440862045 us; error 31 us; skew 500 ppm
Sep 28 23:17:20 dist-test-slave-djkr krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:20.864324 4529 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:17:20.864612 4529 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:20.864660 4529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:20.864773 4529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:20.865760 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b4f571e40074f2f8db7f79e1ed9ee58"
format_stamp: "Formatted at 2026-09-28 23:17:20 on dist-test-slave-djkr"
server_key: "0444cce7c202ebf4e551ebb00f23cbf6"
server_key_iv: "40c0170c031168fd22de82e7aeea0544"
server_key_version: "encryptionkey@0"
I20260928 23:17:20.866071 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b4f571e40074f2f8db7f79e1ed9ee58"
format_stamp: "Formatted at 2026-09-28 23:17:20 on dist-test-slave-djkr"
server_key: "0444cce7c202ebf4e551ebb00f23cbf6"
server_key_iv: "40c0170c031168fd22de82e7aeea0544"
server_key_version: "encryptionkey@0"
I20260928 23:17:20.867149 4529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:17:20.867771 4543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:20.867916 4529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:20.867990 4529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "0b4f571e40074f2f8db7f79e1ed9ee58"
format_stamp: "Formatted at 2026-09-28 23:17:20 on dist-test-slave-djkr"
server_key: "0444cce7c202ebf4e551ebb00f23cbf6"
server_key_iv: "40c0170c031168fd22de82e7aeea0544"
server_key_version: "encryptionkey@0"
I20260928 23:17:20.868045 4529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:20.889410 4529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:20.890936 4529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:20.891033 4529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:20.891183 4529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:20.891412 4529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:20.891440 4529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:20.891458 4529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:20.891475 4529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:20.896546 4529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37495
I20260928 23:17:20.896607 4656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37495 every 8 connection(s)
I20260928 23:17:20.896811 4529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:20.898543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 4529
I20260928 23:17:20.898623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:20.898768 392 external_mini_cluster.cc:1428] Setting key 2e6ee6cde828c1decf7bc19a2509e1dc
Sep 28 23:17:20 dist-test-slave-djkr krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:20.906041 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:20.897392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51757 (local address 127.0.98.62:44595)
0928 23:17:20.897456 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:20.897458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:20.897630 (+ 172us) server_negotiation.cc:407] Connection header received
0928 23:17:20.898063 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:20.898064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:20.898076 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:20.898121 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:20.899039 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:20.899712 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:20.901015 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:20.901315 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:20.901389 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:20.903363 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:20.903365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:20.903365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:20.903375 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:20.904812 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:20.905181 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:20.905183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:20.905183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:20.905241 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:20.905437 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:20.905438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:20.905439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:20.905576 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:20.905644 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:20.905873 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 23:17:20.905951 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:17:20.906088 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:20.897374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44595 (local address 127.0.98.1:51757)
0928 23:17:20.897523 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:20.897557 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:20.897991 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:20.898189 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:20.898192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:20.898421 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:20.898944 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:20.898949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:20.899800 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:20.899801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:20.900939 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:20.900942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:20.901400 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:20.901402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:20.901503 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:20.901945 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:20.901954 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:20.903263 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:20.904897 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:20.904902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:20.904904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:20.905103 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:20.905298 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:20.905299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:20.905299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:20.905385 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:20.905639 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:20.905644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:20.905753 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 23:17:20.905829 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:17:20.905938 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:17:20.906517 4657 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44595
I20260928 23:17:20.906610 4657 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:20.906749 4657 heartbeater.cc:511] Master 127.0.98.62:44595 requested a full tablet report, sending...
I20260928 23:17:20.907207 4467 ts_manager.cc:194] Registered new tserver with Master: 0b4f571e40074f2f8db7f79e1ed9ee58 (127.0.98.1:37495)
I20260928 23:17:20.907874 4467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:17:20.920480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:20.997465 4662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:20.997641 4662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:20.997663 4662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:20.998728 4662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:20.998760 4662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:20.998785 4662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:21.000195 4662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:21.000371 4662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:21.000535 4662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:21.004980 4670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:21.005133 4668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:21.005133 4662 server_base.cc:1034] running on GCE node
W20260928 23:17:21.005237 4667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:21.005468 4662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:21.005693 4662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:21.006848 4662 hybrid_clock.cc:648] HybridClock initialized: now 1790637441006834 us; error 23 us; skew 500 ppm
Sep 28 23:17:21 dist-test-slave-djkr krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:21.009246 4662 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:17:21.009541 4662 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:21.009589 4662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:21.009720 4662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:21.010784 4662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "63526e7ac42a428798d5b7ed5b2710a8"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "0429de5266e550a3bcdf10275610c8af"
server_key_iv: "94057126963c7e3db197d062b623fee5"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.011090 4662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "63526e7ac42a428798d5b7ed5b2710a8"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "0429de5266e550a3bcdf10275610c8af"
server_key_iv: "94057126963c7e3db197d062b623fee5"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.012331 4662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:21.012977 4676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.013116 4662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:21.013199 4662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "63526e7ac42a428798d5b7ed5b2710a8"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "0429de5266e550a3bcdf10275610c8af"
server_key_iv: "94057126963c7e3db197d062b623fee5"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.013263 4662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:21.020867 4662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:21.022549 4662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:21.022670 4662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:21.022840 4662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:21.023092 4662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:21.023123 4662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.023157 4662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:21.023180 4662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.028504 4662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39827
I20260928 23:17:21.028551 4789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39827 every 8 connection(s)
I20260928 23:17:21.028767 4662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:17:21 dist-test-slave-djkr krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:21.034240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 4662
I20260928 23:17:21.034329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:21.034508 392 external_mini_cluster.cc:1428] Setting key 2e03f4784ccf7a8996f53a0d7c3ae285
I20260928 23:17:21.037930 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.029320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36679 (local address 127.0.98.62:44595)
0928 23:17:21.029368 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.029369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.029531 (+ 162us) server_negotiation.cc:407] Connection header received
0928 23:17:21.030002 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.030004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.030017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.030067 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.030908 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.031457 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.032526 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.032738 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.032793 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.034692 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.034694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.034695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:21.034702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.036685 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:21.037043 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:21.037045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:21.037045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:21.037107 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:21.037316 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:21.037317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:21.037317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:21.037442 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.037492 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.037799 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.037868 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:17:21.038117 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.029283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44595 (local address 127.0.98.2:36679)
0928 23:17:21.029425 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.029458 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.029912 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.030131 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.030134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.030366 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:21.030814 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.030819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.031538 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.031539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.032451 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.032453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.032803 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.032804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.032908 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.033262 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:21.033270 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:21.034602 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:21.036765 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:21.036771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:21.036773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:21.036974 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:21.037163 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:21.037163 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:21.037163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:21.037270 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:21.037506 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:21.037511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:21.037657 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.037771 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.037916 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
I20260928 23:17:21.038579 4790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44595
I20260928 23:17:21.038674 4790 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:21.038829 4790 heartbeater.cc:511] Master 127.0.98.62:44595 requested a full tablet report, sending...
I20260928 23:17:21.039175 4467 ts_manager.cc:194] Registered new tserver with Master: 63526e7ac42a428798d5b7ed5b2710a8 (127.0.98.2:39827)
I20260928 23:17:21.039525 4467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36679
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:17:21.058225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:21.134873 4795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:21.135048 4795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:21.135070 4795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:21.136158 4795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:21.136193 4795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:21.136217 4795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:21.137598 4795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:21.137758 4795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:21.137917 4795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:21.142468 4800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:21.142529 4801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:21.142503 4803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:21.142652 4795 server_base.cc:1034] running on GCE node
I20260928 23:17:21.142866 4795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:21.143064 4795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:21.144234 4795 hybrid_clock.cc:648] HybridClock initialized: now 1790637441144224 us; error 26 us; skew 500 ppm
Sep 28 23:17:21 dist-test-slave-djkr krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:21.146507 4795 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:17:21.146842 4795 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:21.146894 4795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:21.147011 4795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:21.148042 4795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "74b0a99fa6894653bd859801ff470a9d"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "b157bbabd584fe039b1d8a4324529e25"
server_key_iv: "dca8eee1e24443f5576b7a722faa84e7"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.148355 4795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "74b0a99fa6894653bd859801ff470a9d"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "b157bbabd584fe039b1d8a4324529e25"
server_key_iv: "dca8eee1e24443f5576b7a722faa84e7"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.149477 4795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:21.150104 4809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.150246 4795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:21.150316 4795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "74b0a99fa6894653bd859801ff470a9d"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "b157bbabd584fe039b1d8a4324529e25"
server_key_iv: "dca8eee1e24443f5576b7a722faa84e7"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.150373 4795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:21.167068 4795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:21.168643 4795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:21.168761 4795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:21.168942 4795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:21.169190 4795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:21.169220 4795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.169245 4795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:21.169270 4795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.174430 4795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36827
I20260928 23:17:21.174461 4922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36827 every 8 connection(s)
I20260928 23:17:21.174677 4795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:17:21 dist-test-slave-djkr krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:21.182255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 4795
I20260928 23:17:21.182376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:21.182552 392 external_mini_cluster.cc:1428] Setting key 9b7d9181ffaed429b137a0690e78b40f
I20260928 23:17:21.182864 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.175272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50715 (local address 127.0.98.62:44595)
0928 23:17:21.175319 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.175321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.175513 (+ 192us) server_negotiation.cc:407] Connection header received
0928 23:17:21.175939 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.175940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.175949 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.176001 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.176872 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.177353 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.178261 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.178467 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.178517 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.180330 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.180332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.180332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:21.180339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.181544 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:21.181862 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:21.181863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:21.181863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:21.181923 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:21.182246 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:21.182247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:21.182247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:21.182369 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.182436 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.182691 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.182767 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:17:21.183027 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.175235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44595 (local address 127.0.98.3:50715)
0928 23:17:21.175396 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.175434 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.175865 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.176051 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.176054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.176339 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:21.176799 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.176804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.177415 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.177416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.178186 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.178188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.178523 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.178524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.178608 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.178983 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:21.178990 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:21.180246 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:21.181602 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:21.181605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:21.181606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:21.181782 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:21.182029 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:21.182030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:21.182030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:21.182137 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:21.182445 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:21.182452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:21.182566 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.182657 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.182811 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260928 23:17:21.183444 4923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44595
I20260928 23:17:21.183528 4923 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:21.183674 4923 heartbeater.cc:511] Master 127.0.98.62:44595 requested a full tablet report, sending...
I20260928 23:17:21.184041 4466 ts_manager.cc:194] Registered new tserver with Master: 74b0a99fa6894653bd859801ff470a9d (127.0.98.3:36827)
I20260928 23:17:21.184372 4466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50715
I20260928 23:17:21.194056 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:17:21.200068 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.195183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.0.98.62:44595)
0928 23:17:21.195253 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.195255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.195328 (+ 73us) server_negotiation.cc:407] Connection header received
0928 23:17:21.195364 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.195365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.195372 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.195413 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.195926 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.196389 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.197034 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.197196 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.197233 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.198196 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.198196 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.198197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:21.198201 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.199192 (+ 991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:21.199468 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:21.199468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:21.199468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:21.199507 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:21.199701 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:21.199701 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:21.199701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:21.199816 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.199883 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.199962 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.200013 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:17:21.201514 4466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:21.202935 4466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:21.208078 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.204681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36827 (local address 127.0.0.1:51188)
0928 23:17:21.204779 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.204830 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.204890 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.205315 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.205316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.205318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:21.205350 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.205351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.206228 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.206229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.207005 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.207007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.207775 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.207776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.207889 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.207902 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.207964 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.208008 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260928 23:17:21.208093 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.204830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.0.98.3:36827)
0928 23:17:21.205136 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.205140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.205162 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:17:21.205200 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.205201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.205214 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.205414 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:21.205472 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.206148 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.207077 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.207697 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.207760 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.207812 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.207984 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.208018 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":75,"threads_started":1}
I20260928 23:17:21.208640 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.204684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37495 (local address 127.0.0.1:40620)
0928 23:17:21.204982 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.204991 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.205039 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.205294 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.205295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.205297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:21.205345 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.205347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.206275 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.206276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.207612 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.207614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.208418 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.208419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.208475 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.208480 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.208531 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.208567 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":1,"thread_start_us":50,"threads_started":1}
I20260928 23:17:21.208644 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.204903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.0.98.1:37495)
0928 23:17:21.205046 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.205052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.205062 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:21.205103 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.205105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.205120 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.205208 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:21.205400 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.206189 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.207665 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.208352 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.208410 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.208462 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.208543 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.208577 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260928 23:17:21.208920 4857 tablet_service.cc:1467] Processing CreateTablet for tablet d8d9a751f69842f4ab87c6688dbbf8aa (DEFAULT_TABLE table=test-table [id=095c75b637c445d4ade600c49ad00494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:21.209247 4857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d9a751f69842f4ab87c6688dbbf8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:21.209348 4591 tablet_service.cc:1467] Processing CreateTablet for tablet d8d9a751f69842f4ab87c6688dbbf8aa (DEFAULT_TABLE table=test-table [id=095c75b637c445d4ade600c49ad00494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:21.209617 4591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d9a751f69842f4ab87c6688dbbf8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:21.209865 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.204684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39827 (local address 127.0.0.1:51072)
0928 23:17:21.205048 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.205057 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.205094 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.205281 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.205284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.205286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:21.205341 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.205343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.207525 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.207526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.208634 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.208636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.209584 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.209585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.209694 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.209701 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.209754 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.209799 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":147,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260928 23:17:21.209964 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.204912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.0.98.2:39827)
0928 23:17:21.205082 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.205085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.205094 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:21.205169 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.205171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.205188 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.205257 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:21.206304 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.207369 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.208753 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.209441 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.209497 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.209546 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.209772 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.209825 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260928 23:17:21.212059 4937 tablet_bootstrap.cc:492] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: Bootstrap starting.
I20260928 23:17:21.212085 4938 tablet_bootstrap.cc:492] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58: Bootstrap starting.
I20260928 23:17:21.212309 4724 tablet_service.cc:1467] Processing CreateTablet for tablet d8d9a751f69842f4ab87c6688dbbf8aa (DEFAULT_TABLE table=test-table [id=095c75b637c445d4ade600c49ad00494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:21.212608 4724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d9a751f69842f4ab87c6688dbbf8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:21.212669 4937 tablet_bootstrap.cc:654] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:21.212677 4938 tablet_bootstrap.cc:654] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:21.213119 4938 log.cc:824] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:21.213183 4937 log.cc:824] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:21.213778 4937 tablet_bootstrap.cc:492] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: No bootstrap required, opened a new log
I20260928 23:17:21.213805 4938 tablet_bootstrap.cc:492] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58: No bootstrap required, opened a new log
I20260928 23:17:21.213840 4937 ts_tablet_manager.cc:1397] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:21.213862 4938 ts_tablet_manager.cc:1397] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:21.215235 4937 raft_consensus.cc:348] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b0a99fa6894653bd859801ff470a9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36827 } } peers { permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 } } peers { permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 } }
I20260928 23:17:21.215361 4937 raft_consensus.cc:374] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:21.215392 4937 raft_consensus.cc:729] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b0a99fa6894653bd859801ff470a9d, State: Initialized, Role: FOLLOWER
I20260928 23:17:21.215494 4937 consensus_queue.cc:260] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b0a99fa6894653bd859801ff470a9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36827 } } peers { permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 } } peers { permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 } }
I20260928 23:17:21.215642 4941 tablet_bootstrap.cc:492] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: Bootstrap starting.
I20260928 23:17:21.215778 4937 ts_tablet_manager.cc:1428] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:17:21.215725 4938 raft_consensus.cc:348] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b0a99fa6894653bd859801ff470a9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36827 } } peers { permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 } } peers { permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 } }
I20260928 23:17:21.215902 4938 raft_consensus.cc:374] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:21.215941 4938 raft_consensus.cc:729] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4f571e40074f2f8db7f79e1ed9ee58, State: Initialized, Role: FOLLOWER
I20260928 23:17:21.215960 4923 heartbeater.cc:503] Master 127.0.98.62:44595 was elected leader, sending a full tablet report...
I20260928 23:17:21.216038 4938 consensus_queue.cc:260] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b0a99fa6894653bd859801ff470a9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36827 } } peers { permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 } } peers { permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 } }
I20260928 23:17:21.216265 4941 tablet_bootstrap.cc:654] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:21.216281 4938 ts_tablet_manager.cc:1428] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:17:21.216410 4657 heartbeater.cc:503] Master 127.0.98.62:44595 was elected leader, sending a full tablet report...
I20260928 23:17:21.216806 4941 log.cc:824] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:21.217444 4941 tablet_bootstrap.cc:492] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: No bootstrap required, opened a new log
I20260928 23:17:21.217497 4941 ts_tablet_manager.cc:1397] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:21.218878 4941 raft_consensus.cc:348] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b0a99fa6894653bd859801ff470a9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36827 } } peers { permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 } } peers { permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 } }
I20260928 23:17:21.218988 4941 raft_consensus.cc:374] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:21.219017 4941 raft_consensus.cc:729] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63526e7ac42a428798d5b7ed5b2710a8, State: Initialized, Role: FOLLOWER
I20260928 23:17:21.219115 4941 consensus_queue.cc:260] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b0a99fa6894653bd859801ff470a9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36827 } } peers { permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 } } peers { permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 } }
I20260928 23:17:21.219340 4790 heartbeater.cc:503] Master 127.0.98.62:44595 was elected leader, sending a full tablet report...
I20260928 23:17:21.219326 4941 ts_tablet_manager.cc:1428] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:17:21.264829 4942 raft_consensus.cc:482] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:21.264928 4942 raft_consensus.cc:504] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b0a99fa6894653bd859801ff470a9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36827 } } peers { permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 } } peers { permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 } }
I20260928 23:17:21.265259 4942 leader_election.cc:288] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63526e7ac42a428798d5b7ed5b2710a8 (127.0.98.2:39827), 0b4f571e40074f2f8db7f79e1ed9ee58 (127.0.98.1:37495)
I20260928 23:17:21.267961 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.265480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38689 (local address 127.0.98.2:39827)
0928 23:17:21.265521 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.265523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.265557 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:17:21.265584 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.265585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.265595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.265648 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:21.265791 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.266320 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.267305 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.267672 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.267726 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.267735 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.267861 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.267895 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:17:21.267980 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.265502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51009 (local address 127.0.98.1:37495)
0928 23:17:21.265538 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.265540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.265677 (+ 137us) server_negotiation.cc:407] Connection header received
0928 23:17:21.265704 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.265705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.265718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.265769 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:21.265918 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.266378 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.267295 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.267680 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.267739 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.267748 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.267886 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.267917 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:17:21.267982 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.265420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39827 (local address 127.0.98.3:38689)
0928 23:17:21.265471 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.265481 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.265518 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.265660 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.265661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.265662 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:21.265732 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.265734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.266387 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.266388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.267219 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.267221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.267732 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.267733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.267805 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.267808 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.267846 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.267880 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":43}
I20260928 23:17:21.267997 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.265433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37495 (local address 127.0.98.3:51009)
0928 23:17:21.265594 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.265603 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.265656 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.265784 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.265785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.265786 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:21.265863 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.265864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.266449 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.266450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.267213 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:21.267214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.267748 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.267748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:21.267821 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.267823 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.267873 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.267906 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260928 23:17:21.268306 4744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d9a751f69842f4ab87c6688dbbf8aa" candidate_uuid: "74b0a99fa6894653bd859801ff470a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63526e7ac42a428798d5b7ed5b2710a8" is_pre_election: true
I20260928 23:17:21.268365 4611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d9a751f69842f4ab87c6688dbbf8aa" candidate_uuid: "74b0a99fa6894653bd859801ff470a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" is_pre_election: true
I20260928 23:17:21.268431 4744 raft_consensus.cc:2454] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b0a99fa6894653bd859801ff470a9d in term 0.
I20260928 23:17:21.268502 4611 raft_consensus.cc:2454] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b0a99fa6894653bd859801ff470a9d in term 0.
I20260928 23:17:21.268626 4812 leader_election.cc:302] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63526e7ac42a428798d5b7ed5b2710a8, 74b0a99fa6894653bd859801ff470a9d; no voters:
I20260928 23:17:21.268746 4942 raft_consensus.cc:2781] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:21.268802 4942 raft_consensus.cc:482] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:21.268817 4942 raft_consensus.cc:3037] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:21.269402 4942 raft_consensus.cc:504] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b0a99fa6894653bd859801ff470a9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36827 } } peers { permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 } } peers { permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 } }
I20260928 23:17:21.269501 4942 leader_election.cc:288] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [CANDIDATE]: Term 1 election: Requested vote from peers 63526e7ac42a428798d5b7ed5b2710a8 (127.0.98.2:39827), 0b4f571e40074f2f8db7f79e1ed9ee58 (127.0.98.1:37495)
I20260928 23:17:21.269624 4611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d9a751f69842f4ab87c6688dbbf8aa" candidate_uuid: "74b0a99fa6894653bd859801ff470a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58"
I20260928 23:17:21.269627 4744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d9a751f69842f4ab87c6688dbbf8aa" candidate_uuid: "74b0a99fa6894653bd859801ff470a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63526e7ac42a428798d5b7ed5b2710a8"
I20260928 23:17:21.269692 4611 raft_consensus.cc:3037] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:21.269706 4744 raft_consensus.cc:3037] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:21.270339 4611 raft_consensus.cc:2454] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b0a99fa6894653bd859801ff470a9d in term 1.
I20260928 23:17:21.270380 4744 raft_consensus.cc:2454] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b0a99fa6894653bd859801ff470a9d in term 1.
I20260928 23:17:21.270493 4811 leader_election.cc:302] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4f571e40074f2f8db7f79e1ed9ee58, 74b0a99fa6894653bd859801ff470a9d; no voters:
I20260928 23:17:21.270609 4942 raft_consensus.cc:2781] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:21.270761 4942 raft_consensus.cc:686] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 1 LEADER]: Becoming Leader. State: Replica: 74b0a99fa6894653bd859801ff470a9d, State: Running, Role: LEADER
I20260928 23:17:21.270843 4942 consensus_queue.cc:237] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b0a99fa6894653bd859801ff470a9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36827 } } peers { permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 } } peers { permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 } }
I20260928 23:17:21.271529 4466 catalog_manager.cc:5495] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 74b0a99fa6894653bd859801ff470a9d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "74b0a99fa6894653bd859801ff470a9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b0a99fa6894653bd859801ff470a9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 } health_report { overall_health: UNKNOWN } } }
W20260928 23:17:21.279203 4791 tablet.cc:2369] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:21.282269 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.280141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.0.98.3:36827)
0928 23:17:21.280175 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.280177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.280184 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:21.280210 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.280211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.280227 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.280280 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:21.280437 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.280850 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.281659 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:21.281832 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:21.281872 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:21.281979 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:21.282061 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:21.282111 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.282198 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.282211 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:17:21.283701 4744 raft_consensus.cc:1261] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8 [term 1 FOLLOWER]: Refusing update from remote peer 74b0a99fa6894653bd859801ff470a9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:21.283706 4611 raft_consensus.cc:1261] T d8d9a751f69842f4ab87c6688dbbf8aa P 0b4f571e40074f2f8db7f79e1ed9ee58 [term 1 FOLLOWER]: Refusing update from remote peer 74b0a99fa6894653bd859801ff470a9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:21.283969 4947 consensus_queue.cc:1035] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4f571e40074f2f8db7f79e1ed9ee58" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:21.284085 4942 consensus_queue.cc:1035] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63526e7ac42a428798d5b7ed5b2710a8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:21.286407 4952 mvcc.cc:205] Tried to move back new op lower bound from 7334450959496036352 to 7334450959446007808. Current Snapshot: MvccSnapshot[applied={T|T < 7334450959496036352}]
I20260928 23:17:21.287976 4467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:21.288049 4467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:21.288705 4467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cbd3f1f502542848d9da675e5382a13: Sending DeleteTablet for 3 replicas of tablet d8d9a751f69842f4ab87c6688dbbf8aa
I20260928 23:17:21.289004 4857 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d9a751f69842f4ab87c6688dbbf8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51188
I20260928 23:17:21.289021 4591 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d9a751f69842f4ab87c6688dbbf8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40620
I20260928 23:17:21.289019 4724 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d9a751f69842f4ab87c6688dbbf8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51072
I20260928 23:17:21.289204 4962 tablet_replica.cc:330] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: stopping tablet replica
I20260928 23:17:21.289440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 4529
I20260928 23:17:21.291625 4960 tablet_replica.cc:330] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: stopping tablet replica
I20260928 23:17:21.291723 4960 raft_consensus.cc:2229] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:21.291833 4960 raft_consensus.cc:2258] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:21.291931 4962 raft_consensus.cc:2229] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:21.292033 4962 raft_consensus.cc:2258] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:21.292420 4960 ts_tablet_manager.cc:1905] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:21.292443 4962 ts_tablet_manager.cc:1905] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:21.293946 4962 ts_tablet_manager.cc:1918] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:21.293951 4960 ts_tablet_manager.cc:1918] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:21.294016 4962 log.cc:1196] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/d8d9a751f69842f4ab87c6688dbbf8aa
I20260928 23:17:21.294109 4960 log.cc:1196] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/d8d9a751f69842f4ab87c6688dbbf8aa
I20260928 23:17:21.294270 4962 ts_tablet_manager.cc:1939] T d8d9a751f69842f4ab87c6688dbbf8aa P 63526e7ac42a428798d5b7ed5b2710a8: Deleting consensus metadata
I20260928 23:17:21.294457 4960 ts_tablet_manager.cc:1939] T d8d9a751f69842f4ab87c6688dbbf8aa P 74b0a99fa6894653bd859801ff470a9d: Deleting consensus metadata
I20260928 23:17:21.294646 4453 catalog_manager.cc:4841] TS 63526e7ac42a428798d5b7ed5b2710a8 (127.0.98.2:39827): tablet d8d9a751f69842f4ab87c6688dbbf8aa (table test-table [id=095c75b637c445d4ade600c49ad00494]) successfully deleted
W20260928 23:17:21.294831 4452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37495 (error 108)
I20260928 23:17:21.294871 4451 catalog_manager.cc:4841] TS 74b0a99fa6894653bd859801ff470a9d (127.0.98.3:36827): tablet d8d9a751f69842f4ab87c6688dbbf8aa (table test-table [id=095c75b637c445d4ade600c49ad00494]) successfully deleted
I20260928 23:17:21.295286 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.295164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37495 (local address 127.0.0.1:40628)
0928 23:17:21.295218 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.295255 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 23:17:21.295292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 4662
W20260928 23:17:21.296057 4452 catalog_manager.cc:4584] TS 0b4f571e40074f2f8db7f79e1ed9ee58 (127.0.98.1:37495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8d9a751f69842f4ab87c6688dbbf8aa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37495: connect: Connection refused (error 111)
I20260928 23:17:21.299695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 4795
I20260928 23:17:21.303918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 4436
2026-09-28T23:17:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1742 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T23:17:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:21Z Disabled control of system clock
I20260928 23:17:21.328475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44295
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:17:21.404894 4967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:21.405082 4967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:21.405102 4967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:21.406176 4967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:21.406211 4967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:21.406222 4967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:21.406234 4967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:21.406243 4967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:21.407523 4967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:21.407728 4967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:21.408016 4967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:21.412372 4972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:21.412401 4973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:21.412447 4975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:21.412607 4967 server_base.cc:1034] running on GCE node
I20260928 23:17:21.412825 4967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:21.413082 4967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:21.414232 4967 hybrid_clock.cc:648] HybridClock initialized: now 1790637441414213 us; error 28 us; skew 500 ppm
I20260928 23:17:21.415387 4967 webserver.cc:460] Webserver started at http://127.0.98.62:39551/ using document root <none> and password file <none>
I20260928 23:17:21.415659 4967 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:21.415711 4967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:21.415838 4967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:21.416847 4967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "fd8af6688d30454c84f6e9f0e41a031a"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "e8f140cbfc62d0bf9752304097f2f7c1"
server_key_iv: "f5191856970b5b79863d2a2860f92e82"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.417169 4967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd8af6688d30454c84f6e9f0e41a031a"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "e8f140cbfc62d0bf9752304097f2f7c1"
server_key_iv: "f5191856970b5b79863d2a2860f92e82"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.418351 4967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:21.419021 4981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.419184 4967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:21.419261 4967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "fd8af6688d30454c84f6e9f0e41a031a"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "e8f140cbfc62d0bf9752304097f2f7c1"
server_key_iv: "f5191856970b5b79863d2a2860f92e82"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.419348 4967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:21.422533 4967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:21.422763 4967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:21.422875 4967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:21.426270 4967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44295
I20260928 23:17:21.426301 5033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44295 every 8 connection(s)
I20260928 23:17:21.426574 4967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:21.427141 5034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:21.429672 5034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a: Bootstrap starting.
I20260928 23:17:21.430179 5034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:21.430428 5034 log.cc:824] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:21.430989 5034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a: No bootstrap required, opened a new log
I20260928 23:17:21.431845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 4967
I20260928 23:17:21.431921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:21.432063 392 external_mini_cluster.cc:1428] Setting key c2db6ae1d648fa95bd781a6abdd8ddeb
I20260928 23:17:21.432792 5034 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8af6688d30454c84f6e9f0e41a031a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44295 } }
I20260928 23:17:21.432965 5034 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:21.433008 5034 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8af6688d30454c84f6e9f0e41a031a, State: Initialized, Role: FOLLOWER
I20260928 23:17:21.433125 5034 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8af6688d30454c84f6e9f0e41a031a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44295 } }
I20260928 23:17:21.433198 5034 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:21.433236 5034 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:21.433279 5034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:21.433801 5034 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8af6688d30454c84f6e9f0e41a031a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44295 } }
I20260928 23:17:21.433907 5034 leader_election.cc:302] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd8af6688d30454c84f6e9f0e41a031a; no voters:
I20260928 23:17:21.434072 5034 leader_election.cc:288] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:21.434106 5039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:21.434238 5039 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [term 1 LEADER]: Becoming Leader. State: Replica: fd8af6688d30454c84f6e9f0e41a031a, State: Running, Role: LEADER
I20260928 23:17:21.434307 5034 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:21.434675 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.432532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.0.98.62:44295)
0928 23:17:21.432747 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.432756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.432787 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:17:21.432901 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.432907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.432964 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.433102 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.433753 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.433757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.433757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.433761 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.433804 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.433818 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.433896 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.433956 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.434101 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260928 23:17:21.434320 5039 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8af6688d30454c84f6e9f0e41a031a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44295 } }
I20260928 23:17:21.435660 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd8af6688d30454c84f6e9f0e41a031a. Latest consensus state: current_term: 1 leader_uuid: "fd8af6688d30454c84f6e9f0e41a031a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8af6688d30454c84f6e9f0e41a031a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44295 } } }
I20260928 23:17:21.435665 5040 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd8af6688d30454c84f6e9f0e41a031a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8af6688d30454c84f6e9f0e41a031a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44295 } } }
I20260928 23:17:21.435746 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:21.435761 5040 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a [sys.catalog]: This master's current role is: LEADER
W20260928 23:17:21.436023 5053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:17:21.436074 5053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:17:21.436112 5055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:21.436247 5055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:21.437597 5055 catalog_manager.cc:1347] Generated new cluster ID: 97c3d2972aad461b9e26cc84acf170b0
I20260928 23:17:21.437649 5055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:21.456970 5055 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:21.457144 5055 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:21.463279 5055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a: Generated new TSK 0
I20260928 23:17:21.463460 5055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:17:21.464538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44295
--builtin_ntp_servers=127.0.98.20:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:17:21.541905 5058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:21.542102 5058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:21.542130 5058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:21.543233 5058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:21.543269 5058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:21.543303 5058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:21.544606 5058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:21.544831 5058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:21.545080 5058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:21.550680 5064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:21.550776 5058 server_base.cc:1034] running on GCE node
W20260928 23:17:21.550690 5066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:21.550690 5063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:21.551039 5058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:21.551270 5058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:21.552400 5058 hybrid_clock.cc:648] HybridClock initialized: now 1790637441552387 us; error 28 us; skew 500 ppm
I20260928 23:17:21.556227 5058 webserver.cc:460] Webserver started at http://127.0.98.1:44757/ using document root <none> and password file <none>
I20260928 23:17:21.556447 5058 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:21.556495 5058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:21.556613 5058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:21.557632 5058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a3bea24474849ada3fd65786aa4f552"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "95ad8b495caafb96d07b9d195b72d26d"
server_key_iv: "c067af411db4987a9fc8d441d7ba5d20"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.557962 5058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a3bea24474849ada3fd65786aa4f552"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "95ad8b495caafb96d07b9d195b72d26d"
server_key_iv: "c067af411db4987a9fc8d441d7ba5d20"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.559207 5058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:17:21.559917 5072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.560065 5058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.560144 5058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "9a3bea24474849ada3fd65786aa4f552"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "95ad8b495caafb96d07b9d195b72d26d"
server_key_iv: "c067af411db4987a9fc8d441d7ba5d20"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.560202 5058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:21.563278 5058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:21.563511 5058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:21.563654 5058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:21.563836 5058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:21.564112 5058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:21.564149 5058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.564173 5058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:21.564189 5058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.569366 5058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39155
I20260928 23:17:21.569418 5185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39155 every 8 connection(s)
I20260928 23:17:21.569684 5058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:21.572221 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.570286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51663 (local address 127.0.98.62:44295)
0928 23:17:21.570360 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.570362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.570532 (+ 170us) server_negotiation.cc:407] Connection header received
0928 23:17:21.570945 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.570946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.570951 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.571000 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.572032 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.572034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.572034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.572038 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.572053 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.572057 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.572105 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.572159 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.572170 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:17:21.572389 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.570270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44295 (local address 127.0.98.1:51663)
0928 23:17:21.570428 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.570458 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.570874 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.571050 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.571053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.571061 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:21.571887 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:21.571887 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:21.571914 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:21.571940 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:21.572108 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:21.572111 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:21.572113 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.572157 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.572200 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 23:17:21.572877 5186 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44295
I20260928 23:17:21.572955 5186 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:21.573087 5186 heartbeater.cc:511] Master 127.0.98.62:44295 requested a full tablet report, sending...
I20260928 23:17:21.573535 4998 ts_manager.cc:194] Registered new tserver with Master: 9a3bea24474849ada3fd65786aa4f552 (127.0.98.1:39155)
I20260928 23:17:21.578696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 5058
I20260928 23:17:21.578778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:21.578924 392 external_mini_cluster.cc:1428] Setting key bf87a1637680d1bcfa51b7337158f847
I20260928 23:17:21.579702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44295
--builtin_ntp_servers=127.0.98.20:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:17:21.657483 5189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:21.657667 5189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:21.657687 5189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:21.658787 5189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:21.658823 5189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:21.658847 5189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:21.660149 5189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:21.660360 5189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:21.660611 5189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:21.665166 5195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:21.665254 5197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:21.665359 5189 server_base.cc:1034] running on GCE node
W20260928 23:17:21.665412 5194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:21.665607 5189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:21.665820 5189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:21.666970 5189 hybrid_clock.cc:648] HybridClock initialized: now 1790637441666956 us; error 27 us; skew 500 ppm
I20260928 23:17:21.668124 5189 webserver.cc:460] Webserver started at http://127.0.98.2:33843/ using document root <none> and password file <none>
I20260928 23:17:21.668334 5189 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:21.668382 5189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:21.668505 5189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:21.669490 5189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "29816eba2579434ebfe8f1e93a22fc8e"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "56e72a6f2945dc192d4532c9ea8b74c0"
server_key_iv: "fc964a0f0c3c81216fa54aff63e07886"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.669809 5189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "29816eba2579434ebfe8f1e93a22fc8e"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "56e72a6f2945dc192d4532c9ea8b74c0"
server_key_iv: "fc964a0f0c3c81216fa54aff63e07886"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.671379 5189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:17:21.672238 5203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.672387 5189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:21.672470 5189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "29816eba2579434ebfe8f1e93a22fc8e"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "56e72a6f2945dc192d4532c9ea8b74c0"
server_key_iv: "fc964a0f0c3c81216fa54aff63e07886"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.672539 5189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:21.677155 5189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:21.677436 5189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:21.677560 5189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:21.677742 5189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:21.678004 5189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:21.678033 5189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.678058 5189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:21.678081 5189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.683202 5189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38425
I20260928 23:17:21.683344 5316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38425 every 8 connection(s)
I20260928 23:17:21.683629 5189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:21.684516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 5189
I20260928 23:17:21.684602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:21.684773 392 external_mini_cluster.cc:1428] Setting key 7ccd0045036ff633076f18e3c0a15eea
I20260928 23:17:21.685658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44295
--builtin_ntp_servers=127.0.98.20:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 23:17:21.687065 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.684317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57899 (local address 127.0.98.62:44295)
0928 23:17:21.684439 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.684442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.684629 (+ 187us) server_negotiation.cc:407] Connection header received
0928 23:17:21.685234 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.685236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.685243 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.685273 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.686787 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.686788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.686789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.686793 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.686809 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.686814 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.686867 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.686982 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.686995 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 23:17:21.687455 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.684234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44295 (local address 127.0.98.2:57899)
0928 23:17:21.684481 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.684548 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.685086 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.685471 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.685475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.685485 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:21.686625 (+ 1140us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:21.686626 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:21.686661 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:21.686696 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:21.686892 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:21.686899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:21.686902 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.687192 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.687259 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260928 23:17:21.688335 5317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44295
I20260928 23:17:21.688421 5317 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:21.688565 5317 heartbeater.cc:511] Master 127.0.98.62:44295 requested a full tablet report, sending...
I20260928 23:17:21.688937 4998 ts_manager.cc:194] Registered new tserver with Master: 29816eba2579434ebfe8f1e93a22fc8e (127.0.98.2:38425)
W20260928 23:17:21.771642 5320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:21.771833 5320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:21.771854 5320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:21.772943 5320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:21.772974 5320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:21.772997 5320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:21.774250 5320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:21.774425 5320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:21.774647 5320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:21.779347 5325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:21.779464 5320 server_base.cc:1034] running on GCE node
W20260928 23:17:21.779412 5328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:21.779376 5326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:21.779755 5320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:21.779968 5320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:21.781100 5320 hybrid_clock.cc:648] HybridClock initialized: now 1790637441781089 us; error 25 us; skew 500 ppm
I20260928 23:17:21.782395 5320 webserver.cc:460] Webserver started at http://127.0.98.3:42709/ using document root <none> and password file <none>
I20260928 23:17:21.782614 5320 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:21.782660 5320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:21.782780 5320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:21.783810 5320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "76b3f24668134be8872ea5961c6f639a"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "aefd00a7eab19cded802cdf5dc9ba56b"
server_key_iv: "c51177c64f3b9a76c3ad5c973ee1cc5e"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.784124 5320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "76b3f24668134be8872ea5961c6f639a"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "aefd00a7eab19cded802cdf5dc9ba56b"
server_key_iv: "c51177c64f3b9a76c3ad5c973ee1cc5e"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.785247 5320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:21.785907 5334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.786062 5320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:21.786136 5320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "76b3f24668134be8872ea5961c6f639a"
format_stamp: "Formatted at 2026-09-28 23:17:21 on dist-test-slave-djkr"
server_key: "aefd00a7eab19cded802cdf5dc9ba56b"
server_key_iv: "c51177c64f3b9a76c3ad5c973ee1cc5e"
server_key_version: "encryptionkey@0"
I20260928 23:17:21.786198 5320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:21.789311 5320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:21.789526 5320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:21.789636 5320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:21.789806 5320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:21.790072 5320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:21.790102 5320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.790146 5320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:21.790170 5320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:21.795140 5320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44255
I20260928 23:17:21.795202 5447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44255 every 8 connection(s)
I20260928 23:17:21.795463 5320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:21.797940 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.796096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58193 (local address 127.0.98.62:44295)
0928 23:17:21.796150 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.796153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.796339 (+ 186us) server_negotiation.cc:407] Connection header received
0928 23:17:21.796738 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.796739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.796743 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.796797 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.797779 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.797779 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.797780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.797782 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.797791 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.797794 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.797825 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.797879 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.797890 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:17:21.798102 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.796071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44295 (local address 127.0.98.3:58193)
0928 23:17:21.796225 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.796262 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.796671 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.796844 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.796847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.796854 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:21.797668 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:21.797668 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:21.797694 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:21.797720 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:21.797830 (+ 110us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:21.797832 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:21.797834 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.797881 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.797924 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260928 23:17:21.798488 5448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44295
I20260928 23:17:21.798564 5448 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:21.798666 5448 heartbeater.cc:511] Master 127.0.98.62:44295 requested a full tablet report, sending...
I20260928 23:17:21.798934 4998 ts_manager.cc:194] Registered new tserver with Master: 76b3f24668134be8872ea5961c6f639a (127.0.98.3:44255)
I20260928 23:17:21.800407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 5320
I20260928 23:17:21.800487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:21.800609 392 external_mini_cluster.cc:1428] Setting key 84d72a8dc09bb6f4f228e7dff6b18f41
I20260928 23:17:21.801285 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:17:21.803294 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.802413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.0.98.62:44295)
0928 23:17:21.802466 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.802468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.802552 (+ 84us) server_negotiation.cc:407] Connection header received
0928 23:17:21.802616 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.802616 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.802621 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.802667 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.803104 (+ 437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.803105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.803105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.803109 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.803122 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.803126 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.803170 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.803217 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.803231 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:17:21.804513 4998 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:21.805953 4998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:21.809677 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.807832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44255 (local address 127.0.0.1:51076)
0928 23:17:21.807980 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.808014 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.808063 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.808386 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.808387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.808388 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:21.809302 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:21.809302 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:21.809307 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:21.809314 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:21.809511 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:21.809516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:21.809519 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.809576 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.809589 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260928 23:17:21.809684 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.807828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38425 (local address 127.0.0.1:41786)
0928 23:17:21.807980 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.808014 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.808063 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.808341 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.808346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.808351 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:21.809258 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:21.809258 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:21.809272 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:21.809312 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:21.809569 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:21.809570 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:21.809570 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.809610 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.809619 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":55,"threads_started":1}
I20260928 23:17:21.809719 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.808041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.0.98.2:38425)
0928 23:17:21.808201 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.808206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.808215 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:21.808238 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.808240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.808249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.808332 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.809440 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.809448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.809449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.809454 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.809488 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.809513 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.809567 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.809619 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.809635 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260928 23:17:21.809799 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.807832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39155 (local address 127.0.0.1:43494)
0928 23:17:21.808138 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.808146 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.808177 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.808802 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.808803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.808806 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:21.809451 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:21.809452 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:21.809458 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:21.809470 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:21.809692 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:21.809692 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:21.809693 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.809720 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.809731 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":4,"thread_start_us":71,"threads_started":1}
I20260928 23:17:21.810020 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.808074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.0.98.1:39155)
0928 23:17:21.808675 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.808678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.808686 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:21.808706 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.808707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.808715 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.808780 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.809553 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.809561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.809562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.809566 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.809596 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.809624 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.809899 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.809926 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.809943 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":65,"threads_started":1}
I20260928 23:17:21.810122 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.807928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.0.98.3:44255)
0928 23:17:21.808268 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.808273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.808280 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:21.808300 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.808302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.808309 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.808374 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.809394 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.809400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.809401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.809405 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.809438 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.809459 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.809510 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.810038 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.810053 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":76,"threads_started":1}
I20260928 23:17:21.810421 5251 tablet_service.cc:1467] Processing CreateTablet for tablet 29a15509cafb474da230e3784e7079cd (DEFAULT_TABLE table=test-table [id=3abd4364d0f242a6bffa1eb647fa1e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:21.810549 5120 tablet_service.cc:1467] Processing CreateTablet for tablet 29a15509cafb474da230e3784e7079cd (DEFAULT_TABLE table=test-table [id=3abd4364d0f242a6bffa1eb647fa1e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:21.810627 5382 tablet_service.cc:1467] Processing CreateTablet for tablet 29a15509cafb474da230e3784e7079cd (DEFAULT_TABLE table=test-table [id=3abd4364d0f242a6bffa1eb647fa1e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:21.810755 5251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a15509cafb474da230e3784e7079cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:21.810863 5120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a15509cafb474da230e3784e7079cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:21.810896 5382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a15509cafb474da230e3784e7079cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:21.813259 5462 tablet_bootstrap.cc:492] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552: Bootstrap starting.
I20260928 23:17:21.813797 5463 tablet_bootstrap.cc:492] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: Bootstrap starting.
I20260928 23:17:21.813915 5462 tablet_bootstrap.cc:654] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:21.814280 5462 log.cc:824] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:21.814383 5463 tablet_bootstrap.cc:654] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:21.814646 5463 log.cc:824] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:21.814944 5464 tablet_bootstrap.cc:492] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e: Bootstrap starting.
I20260928 23:17:21.814996 5462 tablet_bootstrap.cc:492] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552: No bootstrap required, opened a new log
I20260928 23:17:21.815044 5462 ts_tablet_manager.cc:1397] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:21.815253 5463 tablet_bootstrap.cc:492] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: No bootstrap required, opened a new log
I20260928 23:17:21.815306 5463 ts_tablet_manager.cc:1397] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:21.815523 5464 tablet_bootstrap.cc:654] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:21.815850 5464 log.cc:824] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:21.817202 5462 raft_consensus.cc:348] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 } } peers { permanent_uuid: "76b3f24668134be8872ea5961c6f639a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } }
I20260928 23:17:21.817334 5462 raft_consensus.cc:374] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:21.817258 5463 raft_consensus.cc:348] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 } } peers { permanent_uuid: "76b3f24668134be8872ea5961c6f639a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } }
I20260928 23:17:21.817356 5462 raft_consensus.cc:729] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3bea24474849ada3fd65786aa4f552, State: Initialized, Role: FOLLOWER
I20260928 23:17:21.817373 5463 raft_consensus.cc:374] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:21.817413 5463 raft_consensus.cc:729] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b3f24668134be8872ea5961c6f639a, State: Initialized, Role: FOLLOWER
I20260928 23:17:21.817471 5462 consensus_queue.cc:260] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 } } peers { permanent_uuid: "76b3f24668134be8872ea5961c6f639a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } }
I20260928 23:17:21.817514 5463 consensus_queue.cc:260] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 } } peers { permanent_uuid: "76b3f24668134be8872ea5961c6f639a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } }
I20260928 23:17:21.817703 5462 ts_tablet_manager.cc:1428] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:17:21.817710 5186 heartbeater.cc:503] Master 127.0.98.62:44295 was elected leader, sending a full tablet report...
I20260928 23:17:21.817736 5463 ts_tablet_manager.cc:1428] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 23:17:21.817752 5448 heartbeater.cc:503] Master 127.0.98.62:44295 was elected leader, sending a full tablet report...
I20260928 23:17:21.818408 5464 tablet_bootstrap.cc:492] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e: No bootstrap required, opened a new log
I20260928 23:17:21.818476 5464 ts_tablet_manager.cc:1397] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
W20260928 23:17:21.820113 5187 tablet.cc:2369] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:21.820425 5464 raft_consensus.cc:348] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 } } peers { permanent_uuid: "76b3f24668134be8872ea5961c6f639a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } }
I20260928 23:17:21.820534 5464 raft_consensus.cc:374] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:21.820561 5464 raft_consensus.cc:729] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29816eba2579434ebfe8f1e93a22fc8e, State: Initialized, Role: FOLLOWER
I20260928 23:17:21.820636 5464 consensus_queue.cc:260] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 } } peers { permanent_uuid: "76b3f24668134be8872ea5961c6f639a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } }
I20260928 23:17:21.820811 5464 ts_tablet_manager.cc:1428] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:17:21.820850 5317 heartbeater.cc:503] Master 127.0.98.62:44295 was elected leader, sending a full tablet report...
I20260928 23:17:21.876858 5469 raft_consensus.cc:482] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:21.877000 5469 raft_consensus.cc:504] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 } } peers { permanent_uuid: "76b3f24668134be8872ea5961c6f639a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } }
I20260928 23:17:21.877350 5469 leader_election.cc:288] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3bea24474849ada3fd65786aa4f552 (127.0.98.1:39155), 29816eba2579434ebfe8f1e93a22fc8e (127.0.98.2:38425)
I20260928 23:17:21.878737 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.877535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38425 (local address 127.0.98.3:49159)
0928 23:17:21.877701 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.877713 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.877777 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.877930 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.877931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.877934 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:21.878409 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:21.878409 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:21.878417 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:21.878428 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:21.878606 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:21.878607 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:21.878608 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.878658 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.878673 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260928 23:17:21.878762 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.877604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49159 (local address 127.0.98.2:38425)
0928 23:17:21.877706 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.877708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.877783 (+ 75us) server_negotiation.cc:407] Connection header received
0928 23:17:21.877833 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.877834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.877840 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.877907 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.878528 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.878530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.878530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.878535 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.878550 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.878555 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.878603 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.878684 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.878697 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 23:17:21.878791 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.877604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52607 (local address 127.0.98.1:39155)
0928 23:17:21.877662 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.877664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.877938 (+ 274us) server_negotiation.cc:407] Connection header received
0928 23:17:21.877972 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.877973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.877979 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.878031 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.878595 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.878596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.878596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.878601 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.878617 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.878621 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.878670 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.878714 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.878727 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:17:21.878846 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.877539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39155 (local address 127.0.98.3:52607)
0928 23:17:21.877864 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.877874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:21.877926 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:21.878033 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:21.878034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:21.878044 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:21.878499 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:21.878500 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:21.878507 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:21.878518 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:21.878670 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:21.878671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:21.878672 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 23:17:21.878714 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:17:21.878726 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":155,"thread_start_us":107,"threads_started":1}
I20260928 23:17:21.879221 5140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a15509cafb474da230e3784e7079cd" candidate_uuid: "76b3f24668134be8872ea5961c6f639a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3bea24474849ada3fd65786aa4f552" is_pre_election: true
I20260928 23:17:21.879212 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a15509cafb474da230e3784e7079cd" candidate_uuid: "76b3f24668134be8872ea5961c6f639a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29816eba2579434ebfe8f1e93a22fc8e" is_pre_election: true
I20260928 23:17:21.879366 5140 raft_consensus.cc:2454] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b3f24668134be8872ea5961c6f639a in term 0.
I20260928 23:17:21.879366 5271 raft_consensus.cc:2454] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76b3f24668134be8872ea5961c6f639a in term 0.
I20260928 23:17:21.879611 5337 leader_election.cc:302] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b3f24668134be8872ea5961c6f639a, 9a3bea24474849ada3fd65786aa4f552; no voters:
I20260928 23:17:21.879743 5469 raft_consensus.cc:2781] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:21.879808 5469 raft_consensus.cc:482] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:21.879827 5469 raft_consensus.cc:3037] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:21.880497 5469 raft_consensus.cc:504] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 } } peers { permanent_uuid: "76b3f24668134be8872ea5961c6f639a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } }
I20260928 23:17:21.880607 5469 leader_election.cc:288] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [CANDIDATE]: Term 1 election: Requested vote from peers 9a3bea24474849ada3fd65786aa4f552 (127.0.98.1:39155), 29816eba2579434ebfe8f1e93a22fc8e (127.0.98.2:38425)
I20260928 23:17:21.880746 5140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a15509cafb474da230e3784e7079cd" candidate_uuid: "76b3f24668134be8872ea5961c6f639a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3bea24474849ada3fd65786aa4f552"
I20260928 23:17:21.880779 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a15509cafb474da230e3784e7079cd" candidate_uuid: "76b3f24668134be8872ea5961c6f639a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29816eba2579434ebfe8f1e93a22fc8e"
I20260928 23:17:21.880851 5271 raft_consensus.cc:3037] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:21.880853 5140 raft_consensus.cc:3037] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:21.881525 5271 raft_consensus.cc:2454] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b3f24668134be8872ea5961c6f639a in term 1.
I20260928 23:17:21.881537 5140 raft_consensus.cc:2454] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76b3f24668134be8872ea5961c6f639a in term 1.
I20260928 23:17:21.881685 5336 leader_election.cc:302] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29816eba2579434ebfe8f1e93a22fc8e, 76b3f24668134be8872ea5961c6f639a; no voters:
I20260928 23:17:21.881764 5469 raft_consensus.cc:2781] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:21.881896 5469 raft_consensus.cc:686] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 1 LEADER]: Becoming Leader. State: Replica: 76b3f24668134be8872ea5961c6f639a, State: Running, Role: LEADER
I20260928 23:17:21.881978 5469 consensus_queue.cc:237] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 } } peers { permanent_uuid: "76b3f24668134be8872ea5961c6f639a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } } peers { permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } }
I20260928 23:17:21.882797 4997 catalog_manager.cc:5495] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a reported cstate change: term changed from 0 to 1, leader changed from <none> to 76b3f24668134be8872ea5961c6f639a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "76b3f24668134be8872ea5961c6f639a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76b3f24668134be8872ea5961c6f639a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } health_report { overall_health: UNKNOWN } } }
I20260928 23:17:21.904376 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.903425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.0.98.3:44255)
0928 23:17:21.903470 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:21.903472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:21.903485 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:17:21.903568 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:21.903569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:21.903582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:21.903625 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:21.904214 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:21.904215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:21.904215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:21.904219 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:21.904232 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:21.904236 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:21.904275 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:21.904308 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 23:17:21.904319 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:17:21.906021 5140 raft_consensus.cc:1261] T 29a15509cafb474da230e3784e7079cd P 9a3bea24474849ada3fd65786aa4f552 [term 1 FOLLOWER]: Refusing update from remote peer 76b3f24668134be8872ea5961c6f639a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:21.906021 5271 raft_consensus.cc:1261] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e [term 1 FOLLOWER]: Refusing update from remote peer 76b3f24668134be8872ea5961c6f639a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:21.906381 5472 consensus_queue.cc:1035] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3bea24474849ada3fd65786aa4f552" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:21.906502 5469 consensus_queue.cc:1035] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [LEADER]: Connected to new peer: Peer: permanent_uuid: "29816eba2579434ebfe8f1e93a22fc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:21.908942 5477 mvcc.cc:205] Tried to move back new op lower bound from 7334450962044633088 to 7334450961949204480. Current Snapshot: MvccSnapshot[applied={T|T < 7334450962044633088}]
I20260928 23:17:21.909230 5476 mvcc.cc:205] Tried to move back new op lower bound from 7334450962044633088 to 7334450961949204480. Current Snapshot: MvccSnapshot[applied={T|T < 7334450962044633088}]
I20260928 23:17:21.910537 4998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:21.910617 4998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:21.911266 4998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd8af6688d30454c84f6e9f0e41a031a: Sending DeleteTablet for 3 replicas of tablet 29a15509cafb474da230e3784e7079cd
I20260928 23:17:21.911588 5120 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a15509cafb474da230e3784e7079cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:21 UTC) from {username='slave'} at 127.0.0.1:43494
I20260928 23:17:21.911648 5251 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a15509cafb474da230e3784e7079cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:21 UTC) from {username='slave'} at 127.0.0.1:41786
I20260928 23:17:21.911777 5382 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a15509cafb474da230e3784e7079cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:21 UTC) from {username='slave'} at 127.0.0.1:51076
I20260928 23:17:21.911847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 5058
I20260928 23:17:21.913234 5486 tablet_replica.cc:330] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: stopping tablet replica
I20260928 23:17:21.913329 5486 raft_consensus.cc:2229] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:21.913408 5486 raft_consensus.cc:2258] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:21.913545 5485 tablet_replica.cc:330] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e: stopping tablet replica
I20260928 23:17:21.913618 5485 raft_consensus.cc:2229] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:21.913699 5485 raft_consensus.cc:2258] T 29a15509cafb474da230e3784e7079cd P 29816eba2579434ebfe8f1e93a22fc8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:21.914083 5486 ts_tablet_manager.cc:1905] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:21.915691 5486 ts_tablet_manager.cc:1918] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:21.915757 5486 log.cc:1196] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/29a15509cafb474da230e3784e7079cd
I20260928 23:17:21.915990 5486 ts_tablet_manager.cc:1939] T 29a15509cafb474da230e3784e7079cd P 76b3f24668134be8872ea5961c6f639a: Deleting consensus metadata
I20260928 23:17:21.916364 4985 catalog_manager.cc:4841] TS 76b3f24668134be8872ea5961c6f639a (127.0.98.3:44255): tablet 29a15509cafb474da230e3784e7079cd (table test-table [id=3abd4364d0f242a6bffa1eb647fa1e94]) successfully deleted
W20260928 23:17:21.917243 4984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39155 (error 108)
I20260928 23:17:21.917469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 5189
I20260928 23:17:21.918012 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.917551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39155 (local address 127.0.0.1:43500)
0928 23:17:21.917937 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.917980 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367}
W20260928 23:17:21.918114 4984 catalog_manager.cc:4584] TS 9a3bea24474849ada3fd65786aa4f552 (127.0.98.1:39155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29a15509cafb474da230e3784e7079cd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39155: connect: Connection refused (error 111)
I20260928 23:17:21.921590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 5320
I20260928 23:17:21.921676 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:21.921588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38425 (local address 127.0.0.1:41792)
0928 23:17:21.921621 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:21.921650 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 23:17:21.926033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 4967
2026-09-28T23:17:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (621 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 23:17:21.944151 392 mini_oidc.cc:152] Starting JWKS server
I20260928 23:17:21.945127 392 webserver.cc:460] Webserver started at https://127.0.0.1:35769/ using document root <none> and password file <none>
I20260928 23:17:21.945220 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:17:21.945348 392 webserver.cc:460] Webserver started at http://127.0.0.1:33777/ using document root <none> and password file <none>
2026-09-28T23:17:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:21Z Disabled control of system clock
I20260928 23:17:21.952257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33753
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33753
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35769/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/testchainca.cert with env {}
W20260928 23:17:22.029851 5493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:22.030043 5493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:22.030066 5493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:22.031262 5493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:22.031306 5493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:22.031317 5493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:22.031328 5493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:22.031339 5493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:17:22.031350 5493 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:17:22.031360 5493 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35769/jwks.json
I20260928 23:17:22.032984 5493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35769/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:22.033222 5493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:22.033427 5493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:22.038034 5498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:22.038031 5499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:22.038031 5501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:22.038110 5493 server_base.cc:1034] running on GCE node
I20260928 23:17:22.038450 5493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:22.038720 5493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:22.039858 5493 hybrid_clock.cc:648] HybridClock initialized: now 1790637442039840 us; error 29 us; skew 500 ppm
I20260928 23:17:22.040944 5493 webserver.cc:460] Webserver started at http://127.0.98.62:45029/ using document root <none> and password file <none>
I20260928 23:17:22.041153 5493 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:22.041201 5493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:22.041320 5493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:22.042325 5493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "d156ad431ce54a03ad1a08733f294e12"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "906b46262de39d477523f27b9a8b17c3"
server_key_iv: "8cbb852bd7f9bfc46e650da2d3524be8"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.042642 5493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "d156ad431ce54a03ad1a08733f294e12"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "906b46262de39d477523f27b9a8b17c3"
server_key_iv: "8cbb852bd7f9bfc46e650da2d3524be8"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.043815 5493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:22.044495 5507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:22.044718 5493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:22.044804 5493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "d156ad431ce54a03ad1a08733f294e12"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "906b46262de39d477523f27b9a8b17c3"
server_key_iv: "8cbb852bd7f9bfc46e650da2d3524be8"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.044859 5493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:22.065272 5493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:22.065575 5493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:22.065699 5493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:22.069182 5493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33753
I20260928 23:17:22.069243 5559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33753 every 8 connection(s)
I20260928 23:17:22.069542 5493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:22.070420 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:22.072646 5560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12: Bootstrap starting.
I20260928 23:17:22.073195 5560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:22.073457 5560 log.cc:824] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:22.074030 5560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12: No bootstrap required, opened a new log
I20260928 23:17:22.075598 5560 raft_consensus.cc:348] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d156ad431ce54a03ad1a08733f294e12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33753 } }
I20260928 23:17:22.075740 5560 raft_consensus.cc:374] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:22.075771 5560 raft_consensus.cc:729] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d156ad431ce54a03ad1a08733f294e12, State: Initialized, Role: FOLLOWER
I20260928 23:17:22.075915 5560 consensus_queue.cc:260] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d156ad431ce54a03ad1a08733f294e12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33753 } }
I20260928 23:17:22.075994 5560 raft_consensus.cc:388] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:22.076018 5560 raft_consensus.cc:482] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:22.076066 5560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:22.076131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 5493
I20260928 23:17:22.076227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:22.076385 392 external_mini_cluster.cc:1428] Setting key ba416c0c07c9b76d5f09d851b0a13de9
I20260928 23:17:22.076650 5560 raft_consensus.cc:504] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d156ad431ce54a03ad1a08733f294e12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33753 } }
I20260928 23:17:22.076756 5560 leader_election.cc:302] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d156ad431ce54a03ad1a08733f294e12; no voters:
I20260928 23:17:22.076949 5560 leader_election.cc:288] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:22.077100 5566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:22.077195 5560 sys_catalog.cc:564] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:22.077214 5566 raft_consensus.cc:686] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [term 1 LEADER]: Becoming Leader. State: Replica: d156ad431ce54a03ad1a08733f294e12, State: Running, Role: LEADER
I20260928 23:17:22.077316 5566 consensus_queue.cc:237] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d156ad431ce54a03ad1a08733f294e12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33753 } }
I20260928 23:17:22.077607 5566 sys_catalog.cc:455] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d156ad431ce54a03ad1a08733f294e12. Latest consensus state: current_term: 1 leader_uuid: "d156ad431ce54a03ad1a08733f294e12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d156ad431ce54a03ad1a08733f294e12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33753 } } }
I20260928 23:17:22.077687 5566 sys_catalog.cc:458] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:22.077617 5567 sys_catalog.cc:455] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d156ad431ce54a03ad1a08733f294e12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d156ad431ce54a03ad1a08733f294e12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33753 } } }
I20260928 23:17:22.077811 5567 sys_catalog.cc:458] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:22.078042 5569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:22.078562 5569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:22.080191 5569 catalog_manager.cc:1347] Generated new cluster ID: 0b98a67843344079a7bad9eb1b4c931c
I20260928 23:17:22.080243 5569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:22.082116 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:22.076805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.0.98.62:33753)
0928 23:17:22.077048 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:22.077058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:22.077092 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:17:22.077205 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:22.077213 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:22.077284 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:22.077662 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:22.077763 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.079119 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.080146 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.080734 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.080850 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.081424 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:22.081428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:22.081428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:22.081663 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:22.081704 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:22.081718 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:22.081786 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:22.081844 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 23:17:22.081921 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":113,"threads_started":1}
I20260928 23:17:22.082995 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:17:22.094472 5569 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:22.094942 5569 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:22.099529 5569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d156ad431ce54a03ad1a08733f294e12: Generated new TSK 0
I20260928 23:17:22.099716 5569 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 23:17:22.120076 5590 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 23:17:22.120150 5590 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 23:17:22.120384 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:22.105143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.0.98.62:33753)
0928 23:17:22.105213 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:22.105215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:22.105305 (+ 90us) server_negotiation.cc:407] Connection header received
0928 23:17:22.105376 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:22.105376 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:22.105388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:22.105460 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:17:22.105577 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.106317 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.107115 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.107443 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.107506 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.107740 (+ 234us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:17:22.120187 (+ 12447us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 23:17:22.120309 (+ 122us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 23:17:22.120437 5564 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 23:17:22.120764 5589 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:22.105073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33753 (local address 127.0.0.1:46762)
0928 23:17:22.105243 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:22.105252 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:22.105303 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:22.105465 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:22.105465 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:22.105467 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:17:22.105519 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:22.105521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.106422 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.106423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:22.107029 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:22.107031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.107517 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.107520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:22.107610 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.107631 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:17:22.120346 (+ 12715us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 23:17:22.120452 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33753: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33753: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 23:17:22.121165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 5493
2026-09-28T23:17:22Z chronyd exiting
I20260928 23:17:22.156013 392 test_util.cc:175] -----------------------------------------------
I20260928 23:17:22.156088 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637417635540-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (212 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 23:17:22.156798 392 mini_oidc.cc:152] Starting JWKS server
I20260928 23:17:22.157473 392 webserver.cc:460] Webserver started at https://127.0.0.1:35269/ using document root <none> and password file <none>
I20260928 23:17:22.157519 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:17:22.157655 392 webserver.cc:460] Webserver started at http://127.0.0.1:38223/ using document root <none> and password file <none>
2026-09-28T23:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:22Z Disabled control of system clock
I20260928 23:17:22.164279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37573
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37573
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35269/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/testchainca.cert with env {}
W20260928 23:17:22.241907 5596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:22.242112 5596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:22.242141 5596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:22.243259 5596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:22.243297 5596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:22.243312 5596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:22.243327 5596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:22.243342 5596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:17:22.243357 5596 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:17:22.243373 5596 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35269/jwks.json
I20260928 23:17:22.244865 5596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37573
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35269/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:22.245098 5596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:22.245307 5596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:22.249789 5602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:22.249826 5604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:22.249822 5596 server_base.cc:1034] running on GCE node
W20260928 23:17:22.250066 5601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:22.250320 5596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:22.250569 5596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:22.251714 5596 hybrid_clock.cc:648] HybridClock initialized: now 1790637442251699 us; error 31 us; skew 500 ppm
I20260928 23:17:22.252868 5596 webserver.cc:460] Webserver started at http://127.0.98.62:34499/ using document root <none> and password file <none>
I20260928 23:17:22.253067 5596 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:22.253110 5596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:22.253254 5596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:22.254261 5596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "116c572c543141c7b8a5bfca423b373a"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "be2c4a5692821716f540ba016b5ca1b6"
server_key_iv: "24f6fb822907caa805ddc6efc1dc8d1b"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.254567 5596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "116c572c543141c7b8a5bfca423b373a"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "be2c4a5692821716f540ba016b5ca1b6"
server_key_iv: "24f6fb822907caa805ddc6efc1dc8d1b"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.255753 5596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:17:22.256397 5610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:22.256533 5596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:22.256601 5596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "116c572c543141c7b8a5bfca423b373a"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "be2c4a5692821716f540ba016b5ca1b6"
server_key_iv: "24f6fb822907caa805ddc6efc1dc8d1b"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.256664 5596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:22.268070 5596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:22.268327 5596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:22.268446 5596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:22.272038 5596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37573
I20260928 23:17:22.272084 5662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37573 every 8 connection(s)
I20260928 23:17:22.272372 5596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:22.273239 5663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:22.275449 5663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a: Bootstrap starting.
I20260928 23:17:22.275998 5663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:22.276237 5663 log.cc:824] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:22.276851 5663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a: No bootstrap required, opened a new log
I20260928 23:17:22.277849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 5596
I20260928 23:17:22.277931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:22.278085 392 external_mini_cluster.cc:1428] Setting key 9406607cb8a83d3cdf6a902b41768b9c
I20260928 23:17:22.278604 5663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c572c543141c7b8a5bfca423b373a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37573 } }
I20260928 23:17:22.278774 5663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:22.278811 5663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116c572c543141c7b8a5bfca423b373a, State: Initialized, Role: FOLLOWER
I20260928 23:17:22.278930 5663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c572c543141c7b8a5bfca423b373a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37573 } }
I20260928 23:17:22.278990 5663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:22.279017 5663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:22.279062 5663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:22.279651 5663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c572c543141c7b8a5bfca423b373a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37573 } }
I20260928 23:17:22.279738 5663 leader_election.cc:302] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 116c572c543141c7b8a5bfca423b373a; no voters:
I20260928 23:17:22.279891 5663 leader_election.cc:288] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:22.279922 5668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:22.280063 5668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [term 1 LEADER]: Becoming Leader. State: Replica: 116c572c543141c7b8a5bfca423b373a, State: Running, Role: LEADER
I20260928 23:17:22.280112 5663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:22.280144 5668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c572c543141c7b8a5bfca423b373a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37573 } }
I20260928 23:17:22.280449 5668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 116c572c543141c7b8a5bfca423b373a. Latest consensus state: current_term: 1 leader_uuid: "116c572c543141c7b8a5bfca423b373a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c572c543141c7b8a5bfca423b373a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37573 } } }
I20260928 23:17:22.280515 5668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:22.280457 5670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "116c572c543141c7b8a5bfca423b373a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116c572c543141c7b8a5bfca423b373a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37573 } } }
I20260928 23:17:22.280548 5670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:22.281538 5673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:22.282039 5673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:22.283514 5673 catalog_manager.cc:1347] Generated new cluster ID: 67150518d4ed4d4fab0e0ffeeacf0530
I20260928 23:17:22.283582 5673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:22.283878 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:22.278595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.0.98.62:37573)
0928 23:17:22.278788 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:22.278795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:22.278820 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:17:22.278937 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:22.278944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:22.279006 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:22.279142 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:22.279323 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.280863 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.281890 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.282348 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.282456 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.283182 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:22.283185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:22.283186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:22.283373 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:22.283414 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:22.283428 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:22.283491 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:22.283596 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 23:17:22.283673 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260928 23:17:22.284727 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:17:22.304188 5673 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:22.304672 5673 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:22.313797 5673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 116c572c543141c7b8a5bfca423b373a: Generated new TSK 0
I20260928 23:17:22.313977 5673 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 23:17:22.338657 5693 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 23:17:22.338721 5693 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 23:17:22.339059 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:22.325187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.0.98.62:37573)
0928 23:17:22.325243 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:22.325245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:22.325327 (+ 82us) server_negotiation.cc:407] Connection header received
0928 23:17:22.325390 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:22.325391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:22.325404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:22.325474 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:17:22.325573 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.326364 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.327101 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.327409 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.327473 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.327596 (+ 123us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:17:22.338851 (+ 11255us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 23:17:22.338990 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 23:17:22.339097 5692 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:22.325110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37573 (local address 127.0.0.1:38590)
0928 23:17:22.325263 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:22.325272 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:22.325323 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:22.325472 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:22.325473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:22.325474 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:17:22.325522 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:22.325524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.326453 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.326454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:22.327027 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:22.327029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.327469 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.327471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:22.327505 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.327510 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:17:22.338952 (+ 11442us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 23:17:22.339031 (+ 79us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37573: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
W20260928 23:17:22.339115 5667 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 23:17:22.339327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 5596
2026-09-28T23:17:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 23:17:22.375942 392 mini_oidc.cc:152] Starting JWKS server
I20260928 23:17:22.376600 392 webserver.cc:460] Webserver started at https://127.0.0.1:44483/ using document root <none> and password file <none>
I20260928 23:17:22.376644 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:17:22.376788 392 webserver.cc:460] Webserver started at http://127.0.0.1:40155/ using document root <none> and password file <none>
2026-09-28T23:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:22Z Disabled control of system clock
I20260928 23:17:22.383934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44853
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44853
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44483/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 23:17:22.461961 5699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:22.462158 5699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:22.462178 5699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:22.463306 5699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:22.463344 5699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:22.463356 5699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:22.463367 5699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:22.463387 5699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:17:22.463410 5699 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:17:22.463424 5699 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44483/jwks.json
I20260928 23:17:22.464742 5699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44483/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:22.464962 5699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:22.465226 5699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:22.469663 5705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:22.469714 5704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:22.469767 5699 server_base.cc:1034] running on GCE node
W20260928 23:17:22.469676 5707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:22.470084 5699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:22.470360 5699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:22.471506 5699 hybrid_clock.cc:648] HybridClock initialized: now 1790637442471491 us; error 26 us; skew 500 ppm
I20260928 23:17:22.476033 5699 webserver.cc:460] Webserver started at http://127.0.98.62:41417/ using document root <none> and password file <none>
I20260928 23:17:22.476264 5699 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:22.476313 5699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:22.476449 5699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:22.477547 5699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "d685ee9352b849cabcf0ce6b87446e68"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "a8d95c2efcdafde55e9eaf494c8105b2"
server_key_iv: "da370276ed2caf7c25f71ae947b471ae"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.477873 5699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "d685ee9352b849cabcf0ce6b87446e68"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "a8d95c2efcdafde55e9eaf494c8105b2"
server_key_iv: "da370276ed2caf7c25f71ae947b471ae"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.479137 5699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:22.479876 5713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:22.480019 5699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:22.480095 5699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "d685ee9352b849cabcf0ce6b87446e68"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "a8d95c2efcdafde55e9eaf494c8105b2"
server_key_iv: "da370276ed2caf7c25f71ae947b471ae"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.480154 5699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:22.502363 5699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:22.502655 5699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:22.502781 5699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:22.506457 5699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44853
I20260928 23:17:22.506494 5765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44853 every 8 connection(s)
I20260928 23:17:22.506814 5699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:22.507507 5766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:22.507743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 5699
I20260928 23:17:22.507820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:22.507972 392 external_mini_cluster.cc:1428] Setting key 82f37604d6f0d7cf74b4856366ab2f98
I20260928 23:17:22.510658 5766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68: Bootstrap starting.
I20260928 23:17:22.511304 5766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:22.511629 5766 log.cc:824] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:22.512296 5766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68: No bootstrap required, opened a new log
I20260928 23:17:22.513290 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:22.508444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.0.98.62:44853)
0928 23:17:22.508623 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:22.508631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:22.508654 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:17:22.509197 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:22.509204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:22.509268 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:22.509381 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:22.509588 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.510445 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.511462 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.511965 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.512056 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.512679 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:22.512683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:22.512683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:22.512841 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:22.512873 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:22.512883 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:22.512927 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:22.513014 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 23:17:22.513075 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260928 23:17:22.514142 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:17:22.514302 5766 raft_consensus.cc:348] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685ee9352b849cabcf0ce6b87446e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44853 } }
I20260928 23:17:22.514439 5766 raft_consensus.cc:374] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:22.514472 5766 raft_consensus.cc:729] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d685ee9352b849cabcf0ce6b87446e68, State: Initialized, Role: FOLLOWER
I20260928 23:17:22.514583 5766 consensus_queue.cc:260] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685ee9352b849cabcf0ce6b87446e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44853 } }
I20260928 23:17:22.514652 5766 raft_consensus.cc:388] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:22.514681 5766 raft_consensus.cc:482] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:22.514726 5766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:22.515285 5766 raft_consensus.cc:504] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685ee9352b849cabcf0ce6b87446e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44853 } }
I20260928 23:17:22.515388 5766 leader_election.cc:302] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d685ee9352b849cabcf0ce6b87446e68; no voters:
I20260928 23:17:22.515540 5766 leader_election.cc:288] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:22.515643 5771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:22.515771 5771 raft_consensus.cc:686] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [term 1 LEADER]: Becoming Leader. State: Replica: d685ee9352b849cabcf0ce6b87446e68, State: Running, Role: LEADER
I20260928 23:17:22.515807 5766 sys_catalog.cc:564] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:22.515856 5771 consensus_queue.cc:237] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685ee9352b849cabcf0ce6b87446e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44853 } }
I20260928 23:17:22.516196 5772 sys_catalog.cc:455] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d685ee9352b849cabcf0ce6b87446e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685ee9352b849cabcf0ce6b87446e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44853 } } }
I20260928 23:17:22.516283 5772 sys_catalog.cc:458] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:22.516455 5778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:22.516188 5773 sys_catalog.cc:455] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d685ee9352b849cabcf0ce6b87446e68. Latest consensus state: current_term: 1 leader_uuid: "d685ee9352b849cabcf0ce6b87446e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685ee9352b849cabcf0ce6b87446e68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44853 } } }
I20260928 23:17:22.516755 5773 sys_catalog.cc:458] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:22.516901 5778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:22.518267 5778 catalog_manager.cc:1347] Generated new cluster ID: 9a1a495841f24f5da914512690708e49
I20260928 23:17:22.518317 5778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:22.556448 5778 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:22.557201 5778 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:22.565940 5778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d685ee9352b849cabcf0ce6b87446e68: Generated new TSK 0
I20260928 23:17:22.566131 5778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:17:22.582623 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:22.579056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.0.98.62:44853)
0928 23:17:22.579110 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:22.579112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:22.579199 (+ 87us) server_negotiation.cc:407] Connection header received
0928 23:17:22.579249 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:22.579250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:22.579262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:22.579347 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:17:22.579436 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.580170 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.580948 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.581240 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.581316 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.581437 (+ 121us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:17:22.582414 (+ 977us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 23:17:22.582552 (+ 138us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50740: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 23:17:22.582690 5769 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50740: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 23:17:22.584707 5795 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:22.578985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44853 (local address 127.0.0.1:50740)
0928 23:17:22.579135 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:22.579145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:22.579201 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:22.579341 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:22.579342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:22.579343 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:17:22.579393 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:22.579394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.580266 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.580266 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:22.580867 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:22.580869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.581310 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.581313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:22.581350 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.581354 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:17:22.582506 (+ 1152us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 23:17:22.584614 (+ 2108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":1939,"thread_start_us":88,"threads_started":1}
I20260928 23:17:22.585038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 5699
E20260928 23:17:22.593606 5796 webserver.cc:533] Webserver: sslize failed:
E20260928 23:17:22.593662 5796 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T23:17:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T23:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:22Z Disabled control of system clock
I20260928 23:17:22.628334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42571
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42571
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/testchainca.cert with env {}
W20260928 23:17:22.704320 5800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:22.704514 5800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:22.704535 5800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:22.705708 5800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:22.705746 5800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:22.705758 5800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:22.705770 5800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:22.705792 5800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:17:22.705816 5800 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:17:22.705834 5800 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 23:17:22.707278 5800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:22.707499 5800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:22.707695 5800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:22.711980 5808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:22.711982 5805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:22.712007 5806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:22.712168 5800 server_base.cc:1034] running on GCE node
I20260928 23:17:22.712442 5800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:22.712705 5800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:22.713832 5800 hybrid_clock.cc:648] HybridClock initialized: now 1790637442713816 us; error 34 us; skew 500 ppm
I20260928 23:17:22.714937 5800 webserver.cc:460] Webserver started at http://127.0.98.62:40733/ using document root <none> and password file <none>
I20260928 23:17:22.715140 5800 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:22.715184 5800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:22.715303 5800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:22.716329 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "1fcce57ccc9d45bca29fc38325d38d30"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "6a3b3e5deab69d57fda830f2db386b6a"
server_key_iv: "8e22ea181917fbe314a53134972766b3"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.716653 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fcce57ccc9d45bca29fc38325d38d30"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "6a3b3e5deab69d57fda830f2db386b6a"
server_key_iv: "8e22ea181917fbe314a53134972766b3"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.717787 5800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:17:22.718405 5814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:22.718557 5800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:22.718634 5800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "1fcce57ccc9d45bca29fc38325d38d30"
format_stamp: "Formatted at 2026-09-28 23:17:22 on dist-test-slave-djkr"
server_key: "6a3b3e5deab69d57fda830f2db386b6a"
server_key_iv: "8e22ea181917fbe314a53134972766b3"
server_key_version: "encryptionkey@0"
I20260928 23:17:22.718699 5800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:22.726257 5800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:22.726501 5800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:22.726613 5800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:22.730031 5800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42571
I20260928 23:17:22.730058 5866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42571 every 8 connection(s)
I20260928 23:17:22.730340 5800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:22.731149 5867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:22.731693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 5800
I20260928 23:17:22.731770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:22.731920 392 external_mini_cluster.cc:1428] Setting key 40111477c09cb77dd7821ad8f1124140
I20260928 23:17:22.733891 5867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30: Bootstrap starting.
I20260928 23:17:22.734531 5867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:22.734802 5867 log.cc:824] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:22.735440 5867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30: No bootstrap required, opened a new log
I20260928 23:17:22.737478 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:22.732314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.0.98.62:42571)
0928 23:17:22.732523 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:22.732532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:22.732563 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:17:22.733056 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:22.733065 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:22.733140 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:22.733283 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:22.733416 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.734695 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.735540 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.736141 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.736262 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.736823 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:22.736828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:22.736828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:22.737018 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:22.737060 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:22.737076 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:22.737134 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:22.737184 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:17:22.737265 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260928 23:17:22.737682 5867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcce57ccc9d45bca29fc38325d38d30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } }
I20260928 23:17:22.737819 5867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:22.737859 5867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fcce57ccc9d45bca29fc38325d38d30, State: Initialized, Role: FOLLOWER
I20260928 23:17:22.737967 5867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcce57ccc9d45bca29fc38325d38d30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } }
I20260928 23:17:22.738039 5867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:22.738066 5867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:22.738109 5867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:22.738386 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:17:22.738704 5867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcce57ccc9d45bca29fc38325d38d30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } }
I20260928 23:17:22.738806 5867 leader_election.cc:302] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fcce57ccc9d45bca29fc38325d38d30; no voters:
I20260928 23:17:22.738941 5867 leader_election.cc:288] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:22.738977 5872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:22.739099 5872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [term 1 LEADER]: Becoming Leader. State: Replica: 1fcce57ccc9d45bca29fc38325d38d30, State: Running, Role: LEADER
I20260928 23:17:22.739159 5867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:22.739187 5872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcce57ccc9d45bca29fc38325d38d30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } }
I20260928 23:17:22.739506 5872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fcce57ccc9d45bca29fc38325d38d30. Latest consensus state: current_term: 1 leader_uuid: "1fcce57ccc9d45bca29fc38325d38d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcce57ccc9d45bca29fc38325d38d30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } } }
I20260928 23:17:22.739602 5872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:22.739513 5874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fcce57ccc9d45bca29fc38325d38d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fcce57ccc9d45bca29fc38325d38d30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42571 } } }
I20260928 23:17:22.739655 5874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:22.739773 5877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:22.740208 5877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:22.741500 5877 catalog_manager.cc:1347] Generated new cluster ID: 17d7fbf459a840ae8c522444364d61aa
I20260928 23:17:22.741549 5877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:22.754227 5877 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:22.754681 5877 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:22.762132 5877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fcce57ccc9d45bca29fc38325d38d30: Generated new TSK 0
I20260928 23:17:22.762302 5877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:17:22.884222 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:22.768196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.0.98.62:42571)
0928 23:17:22.768237 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:22.768240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:22.768333 (+ 93us) server_negotiation.cc:407] Connection header received
0928 23:17:22.768379 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:22.768380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:22.768392 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:22.768466 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:17:22.768545 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.769331 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.770020 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.770325 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.770383 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.770497 (+ 114us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:17:22.884022 (+113525us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 23:17:22.884164 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56220: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 23:17:22.884279 5870 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56220: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 23:17:22.884351 5896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:22.768136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42571 (local address 127.0.0.1:56220)
0928 23:17:22.768271 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:22.768280 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:22.768332 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:22.768464 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:22.768464 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:22.768466 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:17:22.768509 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:22.768511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.769392 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.769393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:22.769958 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:22.769959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:22.770391 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:22.770393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:22.770428 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:22.770432 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:17:22.884159 (+113727us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 23:17:22.884257 (+ 98us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42571: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260928 23:17:22.884653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 5800
2026-09-28T23:17:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (279 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (82 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:23 dist-test-slave-djkr krb5kdc[5908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:23 dist-test-slave-djkr krb5kdc[5908](info): set up 2 sockets
Sep 28 23:17:23 dist-test-slave-djkr krb5kdc[5908](info): commencing operation
krb5kdc: starting...
W20260928 23:17:23.999296 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:24.104233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41985
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:24.180773 5921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:24.180959 5921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:24.180979 5921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:24.182112 5921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:24.182154 5921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:24.182166 5921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:24.182178 5921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:24.182188 5921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:24.183684 5921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:24.183900 5921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:24.184085 5921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:24.188426 5929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:24.188362 5927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:24.188375 5926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:24.188628 5921 server_base.cc:1034] running on GCE node
I20260928 23:17:24.188860 5921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:24.189121 5921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:24.190258 5921 hybrid_clock.cc:648] HybridClock initialized: now 1790637444190241 us; error 29 us; skew 500 ppm
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.192589 5921 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:24.192986 5921 webserver.cc:460] Webserver started at http://127.0.98.62:36755/ using document root <none> and password file <none>
I20260928 23:17:24.193182 5921 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:24.193228 5921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:24.193342 5921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:24.194320 5921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "3d7336fee2a24e82acb1a40e7b1e120c"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "172ae0d3a5c0b7457c8f4b128ae0ac34"
server_key_iv: "db17182126a17a65a367c01eaa14b3ee"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.194634 5921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d7336fee2a24e82acb1a40e7b1e120c"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "172ae0d3a5c0b7457c8f4b128ae0ac34"
server_key_iv: "db17182126a17a65a367c01eaa14b3ee"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.195854 5921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:24.196471 5936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.196604 5921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:24.196697 5921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "3d7336fee2a24e82acb1a40e7b1e120c"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "172ae0d3a5c0b7457c8f4b128ae0ac34"
server_key_iv: "db17182126a17a65a367c01eaa14b3ee"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.196753 5921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:24.207216 5921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:24.208920 5921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:24.209018 5921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:24.212498 5921 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41985
I20260928 23:17:24.212527 5988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41985 every 8 connection(s)
I20260928 23:17:24.212958 5921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:24.213536 5989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:24.215638 5989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c: Bootstrap starting.
I20260928 23:17:24.216162 5989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:24.216409 5989 log.cc:824] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:24.217011 5989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c: No bootstrap required, opened a new log
I20260928 23:17:24.218079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 5921
I20260928 23:17:24.218178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:24.218324 392 external_mini_cluster.cc:1428] Setting key 3d00caf98fea9d6f56a56138a0ca861e
I20260928 23:17:24.218603 5989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7336fee2a24e82acb1a40e7b1e120c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41985 } }
I20260928 23:17:24.218773 5989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:24.218825 5989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7336fee2a24e82acb1a40e7b1e120c, State: Initialized, Role: FOLLOWER
I20260928 23:17:24.218945 5989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7336fee2a24e82acb1a40e7b1e120c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41985 } }
I20260928 23:17:24.219405 5989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:24.219453 5989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:24.219493 5989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:24.220024 5989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7336fee2a24e82acb1a40e7b1e120c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41985 } }
I20260928 23:17:24.220124 5989 leader_election.cc:302] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d7336fee2a24e82acb1a40e7b1e120c; no voters:
I20260928 23:17:24.220260 5989 leader_election.cc:288] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:24.220294 5994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:24.220427 5994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [term 1 LEADER]: Becoming Leader. State: Replica: 3d7336fee2a24e82acb1a40e7b1e120c, State: Running, Role: LEADER
I20260928 23:17:24.220472 5989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:24.220512 5994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7336fee2a24e82acb1a40e7b1e120c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41985 } }
I20260928 23:17:24.220826 5994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d7336fee2a24e82acb1a40e7b1e120c. Latest consensus state: current_term: 1 leader_uuid: "3d7336fee2a24e82acb1a40e7b1e120c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7336fee2a24e82acb1a40e7b1e120c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41985 } } }
I20260928 23:17:24.220880 5996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d7336fee2a24e82acb1a40e7b1e120c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d7336fee2a24e82acb1a40e7b1e120c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41985 } } }
I20260928 23:17:24.220899 5994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:24.220927 5996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:24.221294 6001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:24.221767 6001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:24.227702 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:24.218765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.0.98.62:41985)
0928 23:17:24.218996 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:24.219005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:24.219036 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:17:24.219148 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:24.219155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:24.219211 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:24.219368 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:24.219972 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.221325 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.222337 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.222712 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.222807 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:24.224914 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:24.224917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:24.224918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:24.224929 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:24.226397 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.226694 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.226697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.226697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.226791 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.226957 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.226958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.226958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.227177 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:24.227304 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:24.227397 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 23:17:24.227524 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":123,"threads_started":1}
I20260928 23:17:24.232151 6001 catalog_manager.cc:1347] Generated new cluster ID: 28a84fa95d2d4eda9d469718aff3f2af
I20260928 23:17:24.232195 6001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:24.244246 6001 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:24.244971 6001 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:24.252337 6001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d7336fee2a24e82acb1a40e7b1e120c: Generated new TSK 0
I20260928 23:17:24.252501 6001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:17:24.278194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41985
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:24.355381 6015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:24.355598 6015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:24.355626 6015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:24.356740 6015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:24.356777 6015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:24.356802 6015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:24.358206 6015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:24.358381 6015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:24.358544 6015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:24.363163 6020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:24.363144 6021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:24.363144 6023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:24.363173 6015 server_base.cc:1034] running on GCE node
I20260928 23:17:24.363533 6015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:24.363792 6015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:24.364928 6015 hybrid_clock.cc:648] HybridClock initialized: now 1790637444364911 us; error 32 us; skew 500 ppm
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.367130 6015 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:17:24.367569 6015 webserver.cc:460] Webserver started at http://127.0.98.1:42735/ using document root <none> and password file <none>
I20260928 23:17:24.367780 6015 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:24.367825 6015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:24.367944 6015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:24.368973 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "7d4716beb2b4408384b7c7942c6ba205"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "739f44f66cb177a6461882359a76ee62"
server_key_iv: "5afcd6671c0289f8869d9f4b26f29270"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.369295 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7d4716beb2b4408384b7c7942c6ba205"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "739f44f66cb177a6461882359a76ee62"
server_key_iv: "5afcd6671c0289f8869d9f4b26f29270"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.370430 6015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:17:24.371042 6030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.371203 6015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:24.371291 6015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "7d4716beb2b4408384b7c7942c6ba205"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "739f44f66cb177a6461882359a76ee62"
server_key_iv: "5afcd6671c0289f8869d9f4b26f29270"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.371357 6015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:24.381364 6015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:24.383450 6015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:24.383598 6015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:24.383800 6015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:24.384081 6015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:24.384120 6015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.384148 6015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:24.384166 6015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.389670 6015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33125
I20260928 23:17:24.389730 6143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33125 every 8 connection(s)
I20260928 23:17:24.390019 6015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:24.392107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 6015
I20260928 23:17:24.392194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:24.392360 392 external_mini_cluster.cc:1428] Setting key 59b56edc469b5d8c6c32a81fb05cc448
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:24.399569 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:24.390678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46155 (local address 127.0.98.62:41985)
0928 23:17:24.390726 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:24.390728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:24.390941 (+ 213us) server_negotiation.cc:407] Connection header received
0928 23:17:24.391905 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:24.391906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:24.391917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:24.391991 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:24.392640 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.393293 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.394359 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.394547 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.394598 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:24.396938 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:24.396939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:24.396939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:24.396946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:24.398146 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.398585 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.398586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.398586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.398635 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.398896 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.398898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.398898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.399030 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:24.399093 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:24.399413 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 23:17:24.399481 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:17:24.399808 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:24.390648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41985 (local address 127.0.98.1:46155)
0928 23:17:24.390819 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:24.390859 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:24.391805 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:24.392048 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:24.392051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:24.392351 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:24.392565 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:24.392570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.393365 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.393366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:24.394266 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:24.394269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.394612 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.394613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:24.394747 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:24.395201 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:24.395211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:24.396851 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:24.398222 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:24.398228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:24.398231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:24.398517 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:24.398691 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:24.398691 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:24.398692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:24.398811 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:24.399100 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:24.399105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:24.399266 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 23:17:24.399397 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 23:17:24.399585 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:17:24.400270 6144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41985
I20260928 23:17:24.400365 6144 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:24.400524 6144 heartbeater.cc:511] Master 127.0.98.62:41985 requested a full tablet report, sending...
I20260928 23:17:24.401024 5953 ts_manager.cc:194] Registered new tserver with Master: 7d4716beb2b4408384b7c7942c6ba205 (127.0.98.1:33125)
I20260928 23:17:24.401618 5953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:17:24.413980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41985
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:24.492394 6149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:24.492586 6149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:24.492615 6149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:24.493731 6149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:24.493769 6149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:24.493810 6149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:24.495214 6149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:24.495446 6149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:24.495714 6149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:24.500273 6157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:24.500283 6155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:24.500440 6149 server_base.cc:1034] running on GCE node
W20260928 23:17:24.500342 6154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:24.500698 6149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:24.500921 6149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:24.502075 6149 hybrid_clock.cc:648] HybridClock initialized: now 1790637444502062 us; error 29 us; skew 500 ppm
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.504468 6149 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:17:24.504894 6149 webserver.cc:460] Webserver started at http://127.0.98.2:33115/ using document root <none> and password file <none>
I20260928 23:17:24.505091 6149 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:24.505151 6149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:24.505275 6149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:24.506259 6149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "632b0053951d4465828dfe888940c73c"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "eb866d61aebe9bff4406554dd93e01de"
server_key_iv: "2c9071a74e4a0e970f9280318407fb21"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.506574 6149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "632b0053951d4465828dfe888940c73c"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "eb866d61aebe9bff4406554dd93e01de"
server_key_iv: "2c9071a74e4a0e970f9280318407fb21"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.507863 6149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:24.508594 6164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.508745 6149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:24.508831 6149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "632b0053951d4465828dfe888940c73c"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "eb866d61aebe9bff4406554dd93e01de"
server_key_iv: "2c9071a74e4a0e970f9280318407fb21"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.508899 6149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:24.519698 6149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:24.521605 6149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:24.521726 6149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:24.521924 6149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:24.522194 6149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:24.522224 6149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.522262 6149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:24.522285 6149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.527622 6149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34721
I20260928 23:17:24.527717 6277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34721 every 8 connection(s)
I20260928 23:17:24.527966 6149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:24.528211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 6149
I20260928 23:17:24.528307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:24.528482 392 external_mini_cluster.cc:1428] Setting key c1ac474b8494b1d56e2c7f67f3142bf4
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:17:24.537025 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:24.528673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38739 (local address 127.0.98.62:41985)
0928 23:17:24.528735 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:24.528737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:24.528934 (+ 197us) server_negotiation.cc:407] Connection header received
0928 23:17:24.529959 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:24.529961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:24.529972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:24.530017 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:24.530626 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.531111 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.532155 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.532349 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.532404 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:24.534527 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:24.534528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:24.534529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:24.534535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:24.535831 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.536174 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.536175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.536175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.536227 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.536441 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.536442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.536442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.536549 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:24.536602 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:24.536853 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 23:17:24.536935 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:17:24.537207 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:24.528667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41985 (local address 127.0.98.2:38739)
0928 23:17:24.528816 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:24.528861 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:24.529860 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:24.530102 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:24.530107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:24.530350 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:24.530541 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:24.530545 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.531191 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.531192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:24.532068 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:24.532071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.532416 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.532416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:24.532536 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:24.532983 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:24.532992 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:24.534441 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:24.535899 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:24.535905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:24.535908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:24.536108 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:24.536278 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:24.536279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:24.536279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:24.536390 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:24.536602 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:24.536609 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:24.536739 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 23:17:24.536827 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:17:24.536974 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260928 23:17:24.537657 6278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41985
I20260928 23:17:24.537752 6278 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 23:17:24.537931 6278 heartbeater.cc:511] Master 127.0.98.62:41985 requested a full tablet report, sending...
I20260928 23:17:24.538336 5953 ts_manager.cc:194] Registered new tserver with Master: 632b0053951d4465828dfe888940c73c (127.0.98.2:34721)
I20260928 23:17:24.538692 5953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:17:24.550062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41985
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:24.628958 6283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:24.629155 6283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:24.629175 6283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:24.630280 6283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:24.630319 6283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:24.630352 6283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:24.631799 6283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:24.632000 6283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:24.632194 6283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:24.636900 6291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:24.636920 6283 server_base.cc:1034] running on GCE node
W20260928 23:17:24.636883 6289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:24.636885 6288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:24.637214 6283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:24.637434 6283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:24.638577 6283 hybrid_clock.cc:648] HybridClock initialized: now 1790637444638566 us; error 30 us; skew 500 ppm
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.640910 6283 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:17:24.641345 6283 webserver.cc:460] Webserver started at http://127.0.98.3:39125/ using document root <none> and password file <none>
I20260928 23:17:24.641528 6283 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:24.641569 6283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:24.641683 6283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:24.642678 6283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "2feebf6b13aa44628f4f67ead7a64dbc"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "e4062fea231e500b45101e177cc5027e"
server_key_iv: "7336f4aac9cad8c3ef5c6b4841c3bbeb"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.642998 6283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2feebf6b13aa44628f4f67ead7a64dbc"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "e4062fea231e500b45101e177cc5027e"
server_key_iv: "7336f4aac9cad8c3ef5c6b4841c3bbeb"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.644234 6283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:24.644941 6298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.645078 6283 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.645162 6283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "2feebf6b13aa44628f4f67ead7a64dbc"
format_stamp: "Formatted at 2026-09-28 23:17:24 on dist-test-slave-djkr"
server_key: "e4062fea231e500b45101e177cc5027e"
server_key_iv: "7336f4aac9cad8c3ef5c6b4841c3bbeb"
server_key_version: "encryptionkey@0"
I20260928 23:17:24.645226 6283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:24.668217 6283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:24.670199 6283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:24.670318 6283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:24.670550 6283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:24.670848 6283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:24.670881 6283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.670910 6283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:24.670928 6283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:24.676440 6283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41981
I20260928 23:17:24.676497 6411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41981 every 8 connection(s)
I20260928 23:17:24.676781 6283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:24.684381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 6283
I20260928 23:17:24.684481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:24.684638 392 external_mini_cluster.cc:1428] Setting key ce2c05c009347a216f3a343d56ef2854
I20260928 23:17:24.685537 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:24.677413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58211 (local address 127.0.98.62:41985)
0928 23:17:24.677462 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:24.677464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:24.677649 (+ 185us) server_negotiation.cc:407] Connection header received
0928 23:17:24.678614 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:24.678615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:24.678626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:24.678674 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:24.679296 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.679836 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.680805 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.680986 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.681038 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:24.683012 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:24.683013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:24.683013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:24.683020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:24.684120 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.684525 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.684527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.684527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.684591 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.684816 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.684818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.684818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.684954 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:24.685022 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:24.685361 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 23:17:24.685443 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:17:24.685703 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:24.677373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41985 (local address 127.0.98.3:58211)
0928 23:17:24.677536 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:24.677573 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:24.678514 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:24.678773 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:24.678777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:24.679024 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:24.679209 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:24.679213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.679921 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.679922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:24.680719 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:24.680722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.681050 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.681051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:24.681167 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:24.681578 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:24.681587 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:24.682932 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:24.684183 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:24.684189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:24.684190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:24.684432 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:24.684646 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:24.684647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:24.684647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:24.684760 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:24.685023 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:24.685029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:24.685192 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 23:17:24.685319 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 23:17:24.685473 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260928 23:17:24.686118 6412 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41985
I20260928 23:17:24.686193 6412 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:24.686357 6412 heartbeater.cc:511] Master 127.0.98.62:41985 requested a full tablet report, sending...
I20260928 23:17:24.686664 5953 ts_manager.cc:194] Registered new tserver with Master: 2feebf6b13aa44628f4f67ead7a64dbc (127.0.98.3:41981)
I20260928 23:17:24.687001 5953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58211
I20260928 23:17:24.696179 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/test-admin.keytab.
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.709064 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 23:17:24.710639 6419 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:24.709833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.0.98.1:33125)
0928 23:17:24.710014 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:24.710020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:24.710029 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:24.710053 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:24.710055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:24.710068 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:24.710143 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:24.710561 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60270: BlockingRecv error: recv got EOF from 127.0.0.1:60270 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
W20260928 23:17:24.710672 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.711634 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.712486 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:17:24.719429 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:24.712965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.0.98.1:33125)
0928 23:17:24.713014 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:24.713015 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:24.713024 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:24.713079 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:24.713080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:24.713089 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:24.713133 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:24.713473 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.714191 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.714882 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.715181 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.715236 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:24.717079 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:24.717085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:24.717086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:24.717092 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:24.718408 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.718716 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.718718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.718718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.718766 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.718895 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.718895 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.718896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.719007 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:24.719094 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:24.719260 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 23:17:24.719331 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 23:17:24.721017 6419 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:24.720617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.0.98.1:33125)
0928 23:17:24.720685 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:24.720687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:24.720697 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:24.720716 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:24.720717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:24.720724 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:24.720764 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:24.720977 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60282: BlockingRecv error: recv got EOF from 127.0.0.1:60282 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 23:17:24.721099 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.721952 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.722688 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:17:24.728495 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:24.723107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.0.98.1:33125)
0928 23:17:24.723153 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:24.723154 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:24.723210 (+ 56us) server_negotiation.cc:407] Connection header received
0928 23:17:24.723242 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:24.723243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:24.723251 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:24.723291 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:24.723624 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.724077 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.724692 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.724876 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.724922 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:24.726680 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:24.726681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:24.726681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:24.726686 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:24.727730 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.727995 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.727996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.727996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.728037 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.728154 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.728154 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.728154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.728229 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:24.728287 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:24.728373 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:17:24.728424 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 23:17:24.729676 6419 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:24.729293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.0.98.1:33125)
0928 23:17:24.729325 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:24.729326 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:24.729353 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:17:24.729407 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:24.729408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:24.729414 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:24.729445 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:24.729619 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60310: BlockingRecv error: recv got EOF from 127.0.0.1:60310 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 23:17:24.729746 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637417635540-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.730536 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:24.731329 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:17:24.736990 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:24.731771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.0.98.1:33125)
0928 23:17:24.731804 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:24.731805 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:24.731840 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:17:24.731877 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:24.731878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:24.731884 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:24.731923 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:24.732200 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.732632 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.733274 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:24.733446 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:24.733500 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:24.735191 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:24.735192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:24.735192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:24.735198 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:24.736261 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.736523 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.736524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.736524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.736566 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:24.736698 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:24.736699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:24.736699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:24.736769 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:24.736824 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:24.736893 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 23:17:24.736941 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:17:24.737454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 6015
I20260928 23:17:24.741403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 6149
I20260928 23:17:24.745395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 6283
I20260928 23:17:24.750255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 5921
2026-09-28T23:17:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1699 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[6432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[6432](info): set up 2 sockets
Sep 28 23:17:24 dist-test-slave-djkr krb5kdc[6432](info): commencing operation
krb5kdc: starting...
W20260928 23:17:25.695444 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:25 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 23:17:25.802181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41385
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:25.878408 6445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:25.878594 6445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:25.878623 6445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:25.879773 6445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:25.879812 6445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:25.879830 6445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:25.879845 6445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:25.879863 6445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:25.881278 6445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:25.881497 6445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:25.881716 6445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:25.886090 6453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:25.886204 6450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:25.886116 6451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:25.886312 6445 server_base.cc:1034] running on GCE node
I20260928 23:17:25.886515 6445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:25.886745 6445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:25.887881 6445 hybrid_clock.cc:648] HybridClock initialized: now 1790637445887865 us; error 31 us; skew 500 ppm
Sep 28 23:17:25 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637445, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:25.890071 6445 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 23:17:25.890456 6445 webserver.cc:460] Webserver started at http://127.0.98.62:41157/ using document root <none> and password file <none>
I20260928 23:17:25.890614 6445 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:25.890652 6445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:25.890729 6445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:25.891681 6445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9"
format_stamp: "Formatted at 2026-09-28 23:17:25 on dist-test-slave-djkr"
server_key: "6988a99a65da9bd028b5d04764dd4ad8"
server_key_iv: "8a4d70dfc1dabad9314c94fbf78ccc5a"
server_key_version: "encryptionkey@0"
I20260928 23:17:25.891959 6445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9"
format_stamp: "Formatted at 2026-09-28 23:17:25 on dist-test-slave-djkr"
server_key: "6988a99a65da9bd028b5d04764dd4ad8"
server_key_iv: "8a4d70dfc1dabad9314c94fbf78ccc5a"
server_key_version: "encryptionkey@0"
I20260928 23:17:25.893069 6445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:25.893687 6460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:25.893822 6445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:25.893898 6445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9"
format_stamp: "Formatted at 2026-09-28 23:17:25 on dist-test-slave-djkr"
server_key: "6988a99a65da9bd028b5d04764dd4ad8"
server_key_iv: "8a4d70dfc1dabad9314c94fbf78ccc5a"
server_key_version: "encryptionkey@0"
I20260928 23:17:25.893955 6445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:25.901420 6445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:25.903066 6445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:25.903189 6445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:25.906795 6445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41385
I20260928 23:17:25.906831 6512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41385 every 8 connection(s)
I20260928 23:17:25.907115 6445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:25.907989 6513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:25.909981 6513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9: Bootstrap starting.
I20260928 23:17:25.910457 6513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:25.910671 6513 log.cc:824] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:25.911237 6513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9: No bootstrap required, opened a new log
I20260928 23:17:25.912755 6513 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } }
I20260928 23:17:25.912881 6513 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:25.912901 6513 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e2b9ff9c38452fa792ffb0dfde9db9, State: Initialized, Role: FOLLOWER
I20260928 23:17:25.913010 6513 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } }
I20260928 23:17:25.913080 6513 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:25.913101 6513 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:25.913132 6513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:25.913815 6513 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } }
I20260928 23:17:25.913892 6513 leader_election.cc:302] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8e2b9ff9c38452fa792ffb0dfde9db9; no voters:
I20260928 23:17:25.914002 6513 leader_election.cc:288] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:25.914050 6516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:25.914206 6513 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:25.914225 6516 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 1 LEADER]: Becoming Leader. State: Replica: c8e2b9ff9c38452fa792ffb0dfde9db9, State: Running, Role: LEADER
I20260928 23:17:25.914321 6516 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } }
I20260928 23:17:25.914682 6518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e2b9ff9c38452fa792ffb0dfde9db9. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } }
I20260928 23:17:25.914773 6518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:25.914945 6525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:25.914660 6517 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } }
I20260928 23:17:25.915180 6517 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:25.915431 6525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:25.916137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 6445
I20260928 23:17:25.916224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:25.916396 392 external_mini_cluster.cc:1428] Setting key 43a283b04ff0b1fa029ffa6d4ef760f2
I20260928 23:17:25.916904 6525 catalog_manager.cc:1347] Generated new cluster ID: be543387447b4543ac1913c407ab356c
I20260928 23:17:25.916947 6525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:17:25 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:25.927070 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:25.916845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.0.98.62:41385)
0928 23:17:25.916968 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:25.916978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:25.917009 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:17:25.917138 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:25.917144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:25.917194 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:25.917329 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:25.917963 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:25.919387 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:25.920396 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:25.920986 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:25.921094 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:25.923660 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:25.923664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:25.923665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:25.923676 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:25.925511 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:25.925862 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:25.925865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:25.925865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:25.925962 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:25.926127 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:25.926128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:25.926128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:25.926404 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:25.926543 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:25.926615 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:17:25.926849 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":60,"threads_started":1}
I20260928 23:17:25.927160 6525 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:25.927923 6525 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:25.934520 6525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9: Generated new TSK 0
I20260928 23:17:25.934661 6525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 23:17:25.957877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41385
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:26.036028 6539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:26.036224 6539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:26.036278 6539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:26.037408 6539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:26.037446 6539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:26.037470 6539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:26.038859 6539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:26.039064 6539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:26.039268 6539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:26.043807 6544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.043787 6545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.043788 6547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.043833 6539 server_base.cc:1034] running on GCE node
I20260928 23:17:26.044160 6539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:26.044359 6539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:26.045509 6539 hybrid_clock.cc:648] HybridClock initialized: now 1790637446045496 us; error 29 us; skew 500 ppm
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:26.047818 6539 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 23:17:26.048239 6539 webserver.cc:460] Webserver started at http://127.0.98.1:36301/ using document root <none> and password file <none>
I20260928 23:17:26.048429 6539 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:26.048475 6539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:26.048591 6539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:26.049554 6539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "0d3ab73582a544b993df86efe8831463"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "7f7e96ce80d3ea66a73f5aed0fc0111b"
server_key_iv: "4fa8a3105e2a1941637d1398f49a971a"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.049880 6539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0d3ab73582a544b993df86efe8831463"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "7f7e96ce80d3ea66a73f5aed0fc0111b"
server_key_iv: "4fa8a3105e2a1941637d1398f49a971a"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.051075 6539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:17:26.051836 6554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.051975 6539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:26.052054 6539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "0d3ab73582a544b993df86efe8831463"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "7f7e96ce80d3ea66a73f5aed0fc0111b"
server_key_iv: "4fa8a3105e2a1941637d1398f49a971a"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.052120 6539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:26.067907 6539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:26.069855 6539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:26.069984 6539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:26.070173 6539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:26.070451 6539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:26.070483 6539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.070511 6539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:26.070534 6539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.075708 6539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44347
I20260928 23:17:26.075771 6667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44347 every 8 connection(s)
I20260928 23:17:26.076030 6539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:26.082255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 6539
I20260928 23:17:26.082337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:26.082511 392 external_mini_cluster.cc:1428] Setting key 5554bce4aaf9c04c8d1570c725ea3b31
I20260928 23:17:26.084674 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.076712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59361 (local address 127.0.98.62:41385)
0928 23:17:26.076784 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.076786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.076925 (+ 139us) server_negotiation.cc:407] Connection header received
0928 23:17:26.077369 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.077370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.077380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.077430 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.078243 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.078731 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.079669 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.079885 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.079942 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.081825 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.081827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.081827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.081834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.083261 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.083712 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.083713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.083713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.083761 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.084012 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.084013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.084013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.084124 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.084194 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.084473 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.084573 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:17:26.084779 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.076684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.1:59361)
0928 23:17:26.076818 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.076851 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.077294 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.077482 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.077485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.077986 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.078161 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.078166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.078793 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.078793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.079577 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.079580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.079955 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.079956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.080049 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.080469 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.080478 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.081738 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.083334 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.083341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.083343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.083630 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.083813 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.083814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.083814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.083935 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.084190 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.084194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.084342 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.084446 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.084607 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260928 23:17:26.085264 6668 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41385
I20260928 23:17:26.085345 6668 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:26.085495 6668 heartbeater.cc:511] Master 127.0.98.62:41385 requested a full tablet report, sending...
I20260928 23:17:26.085963 6477 ts_manager.cc:194] Registered new tserver with Master: 0d3ab73582a544b993df86efe8831463 (127.0.98.1:44347)
I20260928 23:17:26.086783 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59361
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 23:17:26.104033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41385
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:26.180171 6673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:26.180356 6673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:26.180377 6673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:26.181433 6673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:26.181463 6673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:26.181486 6673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:26.182816 6673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:26.182977 6673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:26.183131 6673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:26.187633 6679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.187758 6673 server_base.cc:1034] running on GCE node
W20260928 23:17:26.187628 6678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.187628 6681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.188038 6673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:26.188264 6673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:26.189410 6673 hybrid_clock.cc:648] HybridClock initialized: now 1790637446189394 us; error 27 us; skew 500 ppm
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:26.191534 6673 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 23:17:26.191953 6673 webserver.cc:460] Webserver started at http://127.0.98.2:34119/ using document root <none> and password file <none>
I20260928 23:17:26.192124 6673 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:26.192160 6673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:26.192250 6673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:26.193152 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "e507ca9b69c740e580572a7ae60c673b"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "01b46b6c3f455513a04da20c58f67b2e"
server_key_iv: "58a79b126723d56180ea7359c05833a3"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.193430 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e507ca9b69c740e580572a7ae60c673b"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "01b46b6c3f455513a04da20c58f67b2e"
server_key_iv: "58a79b126723d56180ea7359c05833a3"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.194537 6673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:17:26.195127 6688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.195315 6673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.195402 6673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "e507ca9b69c740e580572a7ae60c673b"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "01b46b6c3f455513a04da20c58f67b2e"
server_key_iv: "58a79b126723d56180ea7359c05833a3"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.195461 6673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:26.208974 6673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:26.210520 6673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:26.210649 6673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:26.210871 6673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:26.211177 6673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:26.211222 6673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.211249 6673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:26.211272 6673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.216840 6673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33065
I20260928 23:17:26.216877 6801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33065 every 8 connection(s)
I20260928 23:17:26.217140 6673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:26.217849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 6673
I20260928 23:17:26.217936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:26.218102 392 external_mini_cluster.cc:1428] Setting key 2b9e4146156f7f398a67882672dc5104
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:17:26.225992 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.217727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49527 (local address 127.0.98.62:41385)
0928 23:17:26.217812 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.217813 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.217956 (+ 143us) server_negotiation.cc:407] Connection header received
0928 23:17:26.218455 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.218456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.218465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.218506 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.219382 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.219909 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.220835 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.221089 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.221153 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.223072 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.223074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.223074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.223081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.224613 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.224983 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.224985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.224986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.225043 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.225360 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.225361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.225361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.225504 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.225604 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.225816 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.225904 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 23:17:26.226058 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.217706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.2:49527)
0928 23:17:26.217849 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.217886 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.218369 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.218572 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.218575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.219111 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.219295 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.219300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.219985 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.219986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.220761 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.220763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.221174 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.221175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.221278 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.221709 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.221717 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.222991 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.224710 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.224716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.224717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.224912 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.225125 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.225126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.225126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.225269 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.225592 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.225595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.225703 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.225779 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.225890 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260928 23:17:26.226574 6802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41385
I20260928 23:17:26.226658 6802 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:26.226823 6802 heartbeater.cc:511] Master 127.0.98.62:41385 requested a full tablet report, sending...
I20260928 23:17:26.227151 6477 ts_manager.cc:194] Registered new tserver with Master: e507ca9b69c740e580572a7ae60c673b (127.0.98.2:33065)
I20260928 23:17:26.227509 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 23:17:26.239692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41385
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:26.317272 6807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:26.317452 6807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:26.317473 6807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:26.318535 6807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:26.318569 6807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:26.318593 6807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:26.319993 6807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:26.320163 6807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:26.320331 6807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:26.324887 6813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.324971 6812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.324869 6815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.325035 6807 server_base.cc:1034] running on GCE node
I20260928 23:17:26.325229 6807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:26.325443 6807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:26.326565 6807 hybrid_clock.cc:648] HybridClock initialized: now 1790637446326550 us; error 30 us; skew 500 ppm
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:26.328940 6807 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 23:17:26.329357 6807 webserver.cc:460] Webserver started at http://127.0.98.3:37105/ using document root <none> and password file <none>
I20260928 23:17:26.329571 6807 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:26.329617 6807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:26.329742 6807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:26.330698 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "708099f2d6dd4974a8c2d8bd90b42666"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "e0c9a7ca36df0205ca47717b9454a401"
server_key_iv: "0d32571ad6367a0acdefb52ab0416cac"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.331001 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "708099f2d6dd4974a8c2d8bd90b42666"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "e0c9a7ca36df0205ca47717b9454a401"
server_key_iv: "0d32571ad6367a0acdefb52ab0416cac"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.332207 6807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:26.332809 6822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.332955 6807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:26.333024 6807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "708099f2d6dd4974a8c2d8bd90b42666"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "e0c9a7ca36df0205ca47717b9454a401"
server_key_iv: "0d32571ad6367a0acdefb52ab0416cac"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.333078 6807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:26.361037 6807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:26.362792 6807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:26.362923 6807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:26.363127 6807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:26.363404 6807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:26.363436 6807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.363466 6807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:26.363485 6807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.368635 6807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38679
I20260928 23:17:26.368695 6935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38679 every 8 connection(s)
I20260928 23:17:26.368973 6807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:26.373826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 6807
I20260928 23:17:26.373912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:26.374073 392 external_mini_cluster.cc:1428] Setting key cae38de01cf5282fe06d5b51be7e8e2b
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:26.377383 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.369565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42173 (local address 127.0.98.62:41385)
0928 23:17:26.369625 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.369627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.369826 (+ 199us) server_negotiation.cc:407] Connection header received
0928 23:17:26.370300 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.370301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.370311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.370369 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.371240 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.371792 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.372769 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.372959 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.373012 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.375284 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.375285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.375285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.375291 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.376382 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.376718 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.376719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.376719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.376763 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.376930 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.376930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.376930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.377023 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.377069 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.377267 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.377323 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:17:26.377555 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.369545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.3:42173)
0928 23:17:26.369692 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.369744 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.370194 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.370438 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.370442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.370978 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.371156 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.371161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.371882 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.371883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.372682 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.372685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.373044 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.373045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.373149 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.373576 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.373584 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.375208 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.376445 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.376450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.376452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.376649 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.376804 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.376804 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.376804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.376887 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.377065 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.377067 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.377176 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.377249 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.377367 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260928 23:17:26.377964 6936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41385
I20260928 23:17:26.378048 6936 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:26.378216 6936 heartbeater.cc:511] Master 127.0.98.62:41385 requested a full tablet report, sending...
I20260928 23:17:26.378545 6477 ts_manager.cc:194] Registered new tserver with Master: 708099f2d6dd4974a8c2d8bd90b42666 (127.0.98.3:38679)
I20260928 23:17:26.378888 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42173
I20260928 23:17:26.385813 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260928 23:17:26.407872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45471
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41385,127.0.98.61:45471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:26.485682 6941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:26.485895 6941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:26.485924 6941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:26.487165 6941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:26.487206 6941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:26.487226 6941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:26.487241 6941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:26.487265 6941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:26.488740 6941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41385,127.0.98.61:45471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
W20260928 23:17:26.488857 6941 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:41385,127.0.98.61:45471'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 23:17:26.488997 6941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:26.489177 6941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:26.493686 6946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.493669 6949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.493688 6947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.493836 6941 server_base.cc:1034] running on GCE node
I20260928 23:17:26.494068 6941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:26.494356 6941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:26.495505 6941 hybrid_clock.cc:648] HybridClock initialized: now 1790637446495492 us; error 30 us; skew 500 ppm
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:26.498065 6941 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 23:17:26.498548 6941 webserver.cc:460] Webserver started at http://127.0.98.61:41179/ using document root <none> and password file <none>
I20260928 23:17:26.498725 6941 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:26.498769 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:26.498857 6941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:26.500171 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data/instance:
uuid: "a61681d380f941c794e06efb97fffac5"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "aed0ea97b7e1d3b08b87bf00ff6b5167"
server_key_iv: "1360c83b3c6966207913f359a27377c0"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.500514 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal/instance:
uuid: "a61681d380f941c794e06efb97fffac5"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "aed0ea97b7e1d3b08b87bf00ff6b5167"
server_key_iv: "1360c83b3c6966207913f359a27377c0"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.501766 6941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:26.502481 6956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.502660 6941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:26.502743 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal
uuid: "a61681d380f941c794e06efb97fffac5"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "aed0ea97b7e1d3b08b87bf00ff6b5167"
server_key_iv: "1360c83b3c6966207913f359a27377c0"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.502805 6941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:26.518052 6941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:26.519950 6941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:26.520080 6941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:26.523883 6941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45471
I20260928 23:17:26.523914 7008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45471 every 8 connection(s)
I20260928 23:17:26.524250 6941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
I20260928 23:17:26.525195 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:26.526645 7009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:17:26.531735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 6941
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:26.535120 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.527325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.0.98.62:41385)
0928 23:17:26.527381 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.527383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.527575 (+ 192us) server_negotiation.cc:407] Connection header received
0928 23:17:26.528035 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.528036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.528047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.528101 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.528686 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.529214 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.530208 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.530397 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.530458 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.532508 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.532509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.532509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.532517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.533849 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.534218 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.534219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.534219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.534271 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.534491 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.534492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.534492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.534628 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.534684 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.534944 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.535020 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:17:26.535269 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.527303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:59562)
0928 23:17:26.527452 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.527485 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.527949 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.528163 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.528166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.528401 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.528588 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.528593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.529291 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.529291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.530122 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.530124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.530463 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.530464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.530556 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.530973 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.530980 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.532420 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.533930 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.533936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.533939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.534151 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.534336 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.534337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.534337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.534438 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.534685 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.534691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.534831 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.534917 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.535061 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260928 23:17:26.535902 7009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:17:26.540067 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.532313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.0.98.61:45471)
0928 23:17:26.532465 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.532469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.532484 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:17:26.532501 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.532502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.532512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.532574 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.533200 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.533976 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.534664 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.534966 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.535023 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.537484 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.537490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.537490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.537496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.538949 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.539314 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.539317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.539317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.539375 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.539524 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.539525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.539525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.539712 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.539855 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.539899 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.539967 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260928 23:17:26.542733 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.536141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45471 (local address 127.0.0.1:60522)
0928 23:17:26.536177 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.536185 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.536218 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.536425 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.536426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.536557 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.536599 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.536601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.537229 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.537231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.537879 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.537882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.538341 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.538342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.538393 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.538685 (+ 292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.538690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.540087 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.542062 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.542063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.542063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.542203 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.542343 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.542343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.542344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.542384 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.542529 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.542530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.542558 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.542599 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.542661 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 23:17:26.542824 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.536196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.0.98.61:45471)
0928 23:17:26.536343 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.536344 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.536349 (+ 5us) server_negotiation.cc:407] Connection header received
0928 23:17:26.536362 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.536362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.536369 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.536419 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.536662 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.537161 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.537953 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.538266 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.538325 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.540317 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.540318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.540318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.540324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.542000 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.542256 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.542257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.542257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.542303 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.542417 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.542417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.542417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.542492 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.542531 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.542615 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.542699 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260928 23:17:26.543994 7009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Bootstrap starting.
I20260928 23:17:26.544540 7009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:26.544772 7009 log.cc:824] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:26.545452 7009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: No bootstrap required, opened a new log
I20260928 23:17:26.546721 7009 raft_consensus.cc:348] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } }
I20260928 23:17:26.546836 7009 raft_consensus.cc:374] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:26.546857 7009 raft_consensus.cc:729] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61681d380f941c794e06efb97fffac5, State: Initialized, Role: FOLLOWER
I20260928 23:17:26.546963 7009 consensus_queue.cc:260] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } }
I20260928 23:17:26.547223 7009 sys_catalog.cc:564] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:26.547183 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } } }
I20260928 23:17:26.547298 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:17:26.548650 7027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:17:26.548691 7027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:17:26.550146 6941 master_runner.cc:428] Detected that this master a61681d380f941c794e06efb97fffac5 is joining an existing cluster
I20260928 23:17:26.550187 6941 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45471
I20260928 23:17:26.557102 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.551089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.0.98.62:41385)
0928 23:17:26.551131 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.551132 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.551284 (+ 152us) server_negotiation.cc:407] Connection header received
0928 23:17:26.551452 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.551453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.551463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.551511 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.551861 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.552614 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.553399 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.553650 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.553727 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.554662 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.554663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.554663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.554669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.556079 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.556430 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.556431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.556432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.556482 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.556672 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.556673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.556673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.556799 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.556863 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.556989 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.557045 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:17:26.557166 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.551043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:59564)
0928 23:17:26.551216 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.551226 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.551293 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.551524 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.551524 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.551709 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.551771 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.551772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.552692 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.552693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.553331 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.553333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.553732 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.553733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.553800 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.554188 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.554193 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.554573 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.556138 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.556139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.556139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.556374 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.556558 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.556559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.556559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.556618 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.556864 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.556864 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.556908 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.556971 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.557060 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260928 23:17:26.557588 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.551068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.0.98.61:45471)
0928 23:17:26.551163 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.551164 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.551281 (+ 117us) server_negotiation.cc:407] Connection header received
0928 23:17:26.551356 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.551357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.551366 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.551413 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.551799 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.552275 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.553056 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.553436 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.553506 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.554513 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.554514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.554515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.554520 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.556004 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.556461 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.556462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.556462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.556501 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.556711 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.556712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.556712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.556807 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.556862 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.557447 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.557513 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 23:17:26.557853 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.551124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45471 (local address 127.0.0.1:60536)
0928 23:17:26.551233 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.551242 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.551297 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.551414 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.551415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.551626 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.551716 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.551718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.552341 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.552342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.552981 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.552982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.553533 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.553534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.553589 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.554038 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.554042 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.554446 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.556086 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.556087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.556087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.556374 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.556559 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.556559 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.556560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.556659 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.557308 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.557309 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.557347 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.557682 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.557763 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":79,"threads_started":1}
I20260928 23:17:26.560238 7016 raft_consensus.cc:482] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:26.560328 7016 raft_consensus.cc:504] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } }
I20260928 23:17:26.560518 7016 leader_election.cc:288] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8e2b9ff9c38452fa792ffb0dfde9db9 (127.0.98.62:41385)
I20260928 23:17:26.560740 6487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a61681d380f941c794e06efb97fffac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" is_pre_election: true
I20260928 23:17:26.561069 6959 leader_election.cc:302] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a61681d380f941c794e06efb97fffac5; no voters: c8e2b9ff9c38452fa792ffb0dfde9db9
I20260928 23:17:26.561209 7016 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637445, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:17:26.565928 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.559012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45471 (local address 127.0.0.1:60548)
0928 23:17:26.559126 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.559164 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.559225 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.559358 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.559360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.559639 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.559684 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.559686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.560592 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.560593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.561385 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.561387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.561809 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.561810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.561859 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.562249 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.562253 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.563411 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.565003 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.565009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.565011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.565211 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.565392 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.565393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.565393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.565444 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.565658 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.565662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.565706 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.565765 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.565854 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260928 23:17:26.565948 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.559068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.0.98.61:45471)
0928 23:17:26.559098 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.559100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.559221 (+ 121us) server_negotiation.cc:407] Connection header received
0928 23:17:26.559282 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.559282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.559291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.559356 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.559750 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.560473 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.561452 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.561742 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.561803 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.563481 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.563482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.563482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.563488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.564914 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.565282 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.565283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.565283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.565336 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.565494 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.565494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.565495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.565608 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.565668 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.565798 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.565863 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:17:26.566425 6477 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45471: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a61681d380f941c794e06efb97fffac5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45471
}
attrs {
promote: true
}
}
dest_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9"
cas_config_opid_index: -1
I20260928 23:17:26.566697 6477 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } }
I20260928 23:17:26.567121 7037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a61681d380f941c794e06efb97fffac5 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } }
I20260928 23:17:26.567390 6983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:26.568146 6983 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 FOLLOWER]: Refusing update from remote peer c8e2b9ff9c38452fa792ffb0dfde9db9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 23:17:26.568423 7036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:26.570130 6983 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a61681d380f941c794e06efb97fffac5 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } }
I20260928 23:17:26.571053 7037 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45471
I20260928 23:17:26.571143 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e2b9ff9c38452fa792ffb0dfde9db9. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } }
I20260928 23:17:26.571206 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:17:26.571239 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } }
I20260928 23:17:26.571265 6941 master.cc:537] Master@127.0.98.61:45471 shutting down...
I20260928 23:17:26.571290 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:17:26.571236 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } }
I20260928 23:17:26.571430 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:17:26.572331 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } }
I20260928 23:17:26.572546 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:26.573529 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } }
I20260928 23:17:26.573598 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:26.573951 6941 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 LEARNER]: Raft consensus shutting down.
I20260928 23:17:26.574057 6941 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 LEARNER]: Raft consensus is shut down!
I20260928 23:17:26.574095 6941 tablet_replica.cc:330] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: stopping tablet replica
I20260928 23:17:26.586825 6941 master.cc:552] Master@127.0.98.61:45471 shutdown complete.
I20260928 23:17:26.586869 6941 master_runner.cc:305] Clearing existing system tablet
I20260928 23:17:26.586982 6941 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:26.587018 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:26.587606 6941 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:26.588027 7048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.588167 6941 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:26.588217 6941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.588238 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal
uuid: "a61681d380f941c794e06efb97fffac5"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "aed0ea97b7e1d3b08b87bf00ff6b5167"
server_key_iv: "1360c83b3c6966207913f359a27377c0"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.588793 6941 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:26.590174 6941 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 23:17:26.590226 6941 log.cc:1196] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 23:17:26.590415 6941 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Deleting consensus metadata
I20260928 23:17:26.590505 6941 master_runner.cc:315] Copying system tablet from 127.0.98.62:41385
I20260928 23:17:26.591041 6941 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41385
I20260928 23:17:26.596978 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.591301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:59568)
0928 23:17:26.591410 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.591419 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.591476 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.591627 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.591628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.591773 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.591823 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.591825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.592742 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.592743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.593309 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.593311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.593749 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.593750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.593794 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.594234 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.594238 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.594504 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.596089 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.596090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.596091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.596267 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.596449 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.596449 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.596449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.596508 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.596743 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.596744 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.596780 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.596833 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.596905 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":75,"threads_started":1}
I20260928 23:17:26.597019 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.591378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.0.98.62:41385)
0928 23:17:26.591423 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.591425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.591464 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:17:26.591521 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.591522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.591532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.591621 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.591882 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.592668 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.593384 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.593685 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.593753 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.594582 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.594583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.594583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.594590 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.596021 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.596320 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.596321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.596322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.596399 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.596563 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.596564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.596564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.596689 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.596753 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.596858 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.596930 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:17:26.597221 6497 tablet_copy_service.cc:140] P c8e2b9ff9c38452fa792ffb0dfde9db9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a61681d380f941c794e06efb97fffac5 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59568)
I20260928 23:17:26.597298 6497 tablet_copy_service.cc:161] P c8e2b9ff9c38452fa792ffb0dfde9db9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a61681d380f941c794e06efb97fffac5 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59568: session id = a61681d380f941c794e06efb97fffac5-00000000000000000000000000000000
I20260928 23:17:26.598116 6497 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:17:26.598698 6941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 23:17:26.599506 6941 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: tablet copy: Starting download of 0 data blocks...
I20260928 23:17:26.599653 6941 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: tablet copy: Starting download of 1 WAL segments...
I20260928 23:17:26.600497 6941 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:17:26.601192 6497 tablet_copy_service.cc:342] P c8e2b9ff9c38452fa792ffb0dfde9db9: Request end of tablet copy session a61681d380f941c794e06efb97fffac5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59568
I20260928 23:17:26.601248 6497 tablet_copy_service.cc:433] P c8e2b9ff9c38452fa792ffb0dfde9db9: ending tablet copy session a61681d380f941c794e06efb97fffac5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a61681d380f941c794e06efb97fffac5
I20260928 23:17:26.603237 6941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:26.604300 7055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.604379 7056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.604506 6941 server_base.cc:1034] running on GCE node
W20260928 23:17:26.604547 7058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.604663 6941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:26.604791 6941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:26.605902 6941 hybrid_clock.cc:648] HybridClock initialized: now 1790637446605896 us; error 22 us; skew 500 ppm
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:26.607208 6941 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 23:17:26.607453 6941 webserver.cc:460] Webserver started at http://127.0.98.61:43439/ using document root <none> and password file <none>
I20260928 23:17:26.607586 6941 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:26.607618 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:26.608238 6941 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:26.608713 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.608824 6941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.608871 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal
uuid: "a61681d380f941c794e06efb97fffac5"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "aed0ea97b7e1d3b08b87bf00ff6b5167"
server_key_iv: "1360c83b3c6966207913f359a27377c0"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.608922 6941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:26.624903 6941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:26.626355 6941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:26.629285 6941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45471
I20260928 23:17:26.629518 7117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45471 every 8 connection(s)
I20260928 23:17:26.629665 6941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
I20260928 23:17:26.630303 7118 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:17:26.630667 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Bootstrap starting.
I20260928 23:17:26.632277 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:17:26.632457 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Bootstrap complete.
I20260928 23:17:26.632615 7118 raft_consensus.cc:348] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } }
I20260928 23:17:26.632678 7118 raft_consensus.cc:729] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a61681d380f941c794e06efb97fffac5, State: Initialized, Role: LEARNER
I20260928 23:17:26.632710 7118 consensus_queue.cc:260] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } }
I20260928 23:17:26.632884 7118 sys_catalog.cc:564] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:26.632874 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } }
I20260928 23:17:26.632956 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:17:26.634091 7132 catalog_manager.cc:1259] Loaded cluster ID: be543387447b4543ac1913c407ab356c
I20260928 23:17:26.634130 7132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: loading cluster ID for follower catalog manager: success
I20260928 23:17:26.635062 7132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: acquiring CA information for follower catalog manager: success
I20260928 23:17:26.635319 7132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:17:26.647851 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.641478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.98.61:45471)
0928 23:17:26.641593 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.641595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.641605 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:26.641628 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.641629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.641640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.641721 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.642189 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.643195 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.643915 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.644215 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.644309 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.645372 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.645373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.645374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.645381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.646822 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.647108 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.647109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.647109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.647170 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.647319 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.647320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.647321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.647443 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.647590 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.647652 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.647731 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260928 23:17:26.669879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37281
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41385,127.0.98.61:45471,127.0.98.60:37281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:26.748946 7136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:26.749151 7136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:26.749171 7136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:26.750278 7136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:26.750319 7136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:26.750332 7136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:26.750344 7136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:26.750353 7136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:26.751820 7136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41385,127.0.98.61:45471,127.0.98.60:37281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:26.752032 7136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:26.752251 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:26.756814 7142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.756841 7141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.756919 7136 server_base.cc:1034] running on GCE node
W20260928 23:17:26.756768 7144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.757226 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:26.757501 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:26.758683 7136 hybrid_clock.cc:648] HybridClock initialized: now 1790637446758667 us; error 26 us; skew 500 ppm
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:26.761534 7136 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 23:17:26.761969 7136 webserver.cc:460] Webserver started at http://127.0.98.60:36151/ using document root <none> and password file <none>
I20260928 23:17:26.762173 7136 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:26.762219 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:26.762343 7136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:26.763363 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data/instance:
uuid: "cff6d43cf03a4cf7b839ed277bd30c33"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "1be3329fb4dfd931e25fa9275828aecd"
server_key_iv: "d11d0a27af18e9d2c3f9f6d471f04ebf"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.763708 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal/instance:
uuid: "cff6d43cf03a4cf7b839ed277bd30c33"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "1be3329fb4dfd931e25fa9275828aecd"
server_key_iv: "d11d0a27af18e9d2c3f9f6d471f04ebf"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.764878 7136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:26.765511 7151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.765650 7136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:26.765725 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal
uuid: "cff6d43cf03a4cf7b839ed277bd30c33"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "1be3329fb4dfd931e25fa9275828aecd"
server_key_iv: "d11d0a27af18e9d2c3f9f6d471f04ebf"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.765785 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:26.777006 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:26.778995 7136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:26.779124 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:26.782971 7136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37281
I20260928 23:17:26.783000 7203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37281 every 8 connection(s)
I20260928 23:17:26.783305 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
I20260928 23:17:26.783974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 7136
I20260928 23:17:26.784802 7204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:26.786319 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:26.793524 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.787084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.0.98.62:41385)
0928 23:17:26.787153 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.787156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.787165 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:26.787189 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.787190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.787201 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.787249 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.787702 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.788169 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.789017 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.789334 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.789411 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.791281 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.791283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.791283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.791289 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.792452 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.792778 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.792780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.792780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.792835 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.793022 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.793023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.793023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.793130 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.793186 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.793360 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.793438 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:17:26.793725 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.786827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:59584)
0928 23:17:26.786957 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.786992 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.787051 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.787269 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.787272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.787521 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.787627 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.787629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.788244 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.788244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.788932 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.788934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.789414 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.789415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.789481 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.789915 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.789924 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.791206 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.792524 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.792528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.792530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.792714 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.792885 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.792886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.792886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.792968 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.793182 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.793187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.793251 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.793326 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.793477 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260928 23:17:26.793726 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.784535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.0.98.60:37281)
0928 23:17:26.784712 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.784723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.784754 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:17:26.785237 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.785244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.785291 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.785413 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.785968 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.786908 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.787797 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.788202 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.788293 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.790595 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.790601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.790601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.790606 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.792438 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.792758 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.792760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.792761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.792832 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.792995 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.792995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.792996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.793211 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.793335 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.793421 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.793548 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260928 23:17:26.794210 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:17:26.799480 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.794412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45471 (local address 127.0.0.1:60574)
0928 23:17:26.794452 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.794463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.794501 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.794605 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.794605 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.794706 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.794744 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.794745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.795288 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.795289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.795874 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.795876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.796197 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.796197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.796247 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.796514 (+ 267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.796517 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.797527 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.798757 (+ 1230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.798758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.798758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.798930 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.799081 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.799082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.799082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.799125 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.799287 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.799288 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.799317 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.799361 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.799423 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:17:26.799497 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.794485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.0.98.61:45471)
0928 23:17:26.794528 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.794529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.794535 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:17:26.794554 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.794555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.794565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.794603 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.794783 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.795239 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.795935 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.796142 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.796200 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.797606 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.797607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.797608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.797614 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.798696 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.798991 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.798991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.798992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.799042 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.799158 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.799158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.799159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.799246 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.799287 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.799378 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.799437 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:17:26.799862 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 23:17:26.805207 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.800080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37281 (local address 127.0.0.1:48792)
0928 23:17:26.800116 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.800125 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.800187 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.800275 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.800276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.800383 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.800407 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.800408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.800939 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.800940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.801400 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.801401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.801704 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.801705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.801749 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.802094 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.802098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.803164 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.804428 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.804429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.804429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.804583 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.804737 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.804737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.804737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.804790 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.804957 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.804957 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.804987 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.805030 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.805100 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 23:17:26.805215 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.800108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.0.98.60:37281)
0928 23:17:26.800147 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.800149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.800156 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:26.800224 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.800224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.800231 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.800272 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.800445 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.800884 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.801456 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.801631 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.801675 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.803234 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.803235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.803236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.803241 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.804362 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.804639 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.804640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.804640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.804680 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.804835 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.804836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.804836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.804918 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.804959 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.805045 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.805150 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":25}
I20260928 23:17:26.806416 7204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: Bootstrap starting.
I20260928 23:17:26.807166 7204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:26.807405 7204 log.cc:824] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:26.808125 7204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: No bootstrap required, opened a new log
I20260928 23:17:26.809482 7204 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } }
I20260928 23:17:26.809612 7204 raft_consensus.cc:374] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:26.809645 7204 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff6d43cf03a4cf7b839ed277bd30c33, State: Initialized, Role: FOLLOWER
I20260928 23:17:26.809768 7204 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } }
I20260928 23:17:26.810034 7209 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } } }
I20260928 23:17:26.810081 7204 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:26.810122 7209 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:17:26.811579 7220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:17:26.811645 7220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:17:26.813794 7136 master_runner.cc:428] Detected that this master cff6d43cf03a4cf7b839ed277bd30c33 is joining an existing cluster
I20260928 23:17:26.813831 7136 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:37281
I20260928 23:17:26.821655 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.815567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.0.98.62:41385)
0928 23:17:26.815603 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.815605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.815612 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:26.815633 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.815634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.815643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.815695 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.816112 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.816824 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.817759 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.818034 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.818108 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.819039 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.819040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.819041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.819047 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.820543 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.820923 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.820924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.820924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.820988 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.821172 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.821172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.821173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.821300 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.821361 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.821495 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.821568 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:17:26.821674 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.815190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:59592)
0928 23:17:26.815448 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.815458 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.815514 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.815744 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.815745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.815970 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.816033 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.816035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.816878 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.816879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.817632 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.817634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.818122 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.818123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.818183 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.818580 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.818584 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.818957 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.820618 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.820619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.820620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.820823 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.821048 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.821049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.821049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.821112 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.821359 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.821360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.821402 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.821470 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.821565 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":52,"thread_start_us":61,"threads_started":1}
I20260928 23:17:26.824932 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.815380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37281 (local address 127.0.0.1:48806)
0928 23:17:26.819647 (+ 4267us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.819658 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.819715 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.819882 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.819883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.820060 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.820114 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.820116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.820871 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.820871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.821609 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.821612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.821987 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.821988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.822026 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.822341 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.822346 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.822649 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.824025 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.824026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.824026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.824170 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.824325 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.824325 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.824326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.824375 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.824587 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.824587 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.824634 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.824692 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.824794 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4250,"thread_start_us":56,"threads_started":1}
I20260928 23:17:26.824941 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.815418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.0.98.60:37281)
0928 23:17:26.815459 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.815466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.819720 (+ 4254us) server_negotiation.cc:407] Connection header received
0928 23:17:26.819767 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.819768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.819778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.819876 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.820191 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.820807 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.821686 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.821896 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.821941 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.822714 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.822715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.822716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.822721 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.823957 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.824230 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.824231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.824232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.824276 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.824421 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.824421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.824422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.824534 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.824585 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.824720 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.824841 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":23}
I20260928 23:17:26.843175 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.815237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.0.98.61:45471)
0928 23:17:26.815276 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.815279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.815513 (+ 234us) server_negotiation.cc:407] Connection header received
0928 23:17:26.815626 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.815627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.815638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.815682 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.815989 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.816754 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.817882 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.818159 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.818210 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.840514 (+ 22304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.840515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.840516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.840524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.842063 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.842434 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.842436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.842436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.842500 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.842682 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.842682 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.842683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.842812 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.842871 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.843002 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.843085 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:17:26.843192 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.815297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45471 (local address 127.0.0.1:60580)
0928 23:17:26.815448 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.815458 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.815533 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.815684 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.815685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.815866 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.815923 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.815925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.816831 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.816832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.817797 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.817799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.818221 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.818222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.818299 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.839992 (+ 21693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.839998 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.840418 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.842144 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.842145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.842146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.842360 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.842562 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.842563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.842563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.842622 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.842869 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.842869 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.842913 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.842976 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.843075 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":32,"thread_start_us":62,"threads_started":1}
I20260928 23:17:26.844877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 6539
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637445, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 23:17:26.849987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44347
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41385
--webserver_port=36301
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41385,127.0.98.61:45471
--tserver_master_addrs=127.0.98.62:41385,127.0.98.61:45471,127.0.98.60:37281 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:17:26.851008 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.841010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.0.98.60:37281)
0928 23:17:26.841128 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.841130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.841137 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:26.841170 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.841171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.841179 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.841228 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.841537 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.842231 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.843111 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.843372 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.843429 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.848384 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.848386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.848386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.848393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.849929 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.850285 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.850286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.850286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.850339 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.850544 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.850544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.850545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.850666 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.850722 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.850860 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.850929 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 23:17:26.851038 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.840954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37281 (local address 127.0.0.1:48808)
0928 23:17:26.841017 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.841026 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.841063 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.841241 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.841242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.841415 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.841467 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.841469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.842302 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.842303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.843029 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.843032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.843440 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.843441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.843519 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.843927 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.843933 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.848298 (+ 4365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.850012 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.850013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.850014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.850206 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.850405 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.850406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.850406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.850480 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.850722 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.850723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.850769 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.850835 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.850940 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 23:17:26.851444 6477 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:37281: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 37281
}
attrs {
promote: true
}
}
dest_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9"
cas_config_opid_index: 5
I20260928 23:17:26.851697 6477 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } }
I20260928 23:17:26.852041 7037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cff6d43cf03a4cf7b839ed277bd30c33 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.852928 7037 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:37281
I20260928 23:17:26.852996 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.853050 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:26.853425 7136 master.cc:537] Master@127.0.98.60:37281 shutting down...
I20260928 23:17:26.853446 7178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:26.854445 7178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 FOLLOWER]: Refusing update from remote peer c8e2b9ff9c38452fa792ffb0dfde9db9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 23:17:26.854702 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:17:26.855204 6463 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 23:17:26.855607 6463 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 -> Peer cff6d43cf03a4cf7b839ed277bd30c33 (127.0.98.60:37281): Couldn't send request to peer cff6d43cf03a4cf7b839ed277bd30c33. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 23:17:26.856360 7136 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:26.856451 7136 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:26.856503 7136 tablet_replica.cc:330] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: stopping tablet replica
I20260928 23:17:26.856943 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.852165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45471 (local address 127.0.0.1:60592)
0928 23:17:26.852389 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.852398 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.852441 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.852632 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.852633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.852634 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:26.852669 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.852671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.853719 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.853720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.855073 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.855076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.856683 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.856684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.856754 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.856779 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.856837 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.856879 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260928 23:17:26.857126 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.852246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.0.98.61:45471)
0928 23:17:26.852363 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.852365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.852487 (+ 122us) server_negotiation.cc:407] Connection header received
0928 23:17:26.852508 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.852509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.852532 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.852580 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:26.852733 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.853449 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.855164 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.855773 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.855842 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.855912 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.857025 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.857067 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 23:17:26.857437 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 LEARNER]: Refusing update from remote peer c8e2b9ff9c38452fa792ffb0dfde9db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 23:17:26.857812 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 23:17:26.858230 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cff6d43cf03a4cf7b839ed277bd30c33 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.859203 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.859256 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e2b9ff9c38452fa792ffb0dfde9db9. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: true } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.859366 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:17:26.859382 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:17:26.863978 7037 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9: attempting to promote NON_VOTER a61681d380f941c794e06efb97fffac5 to VOTER
I20260928 23:17:26.864183 7037 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } }
I20260928 23:17:26.864604 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.864505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37281 (local address 127.0.0.1:48810)
0928 23:17:26.864542 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.864576 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:17:26.864661 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 LEARNER]: Refusing update from remote peer c8e2b9ff9c38452fa792ffb0dfde9db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 23:17:26.864822 7036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 23:17:26.865341 7045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a61681d380f941c794e06efb97fffac5 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.866214 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.866283 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:26.866343 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.866384 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:26.866533 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a61681d380f941c794e06efb97fffac5 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.867342 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.867414 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.865210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37281 (local address 127.0.0.1:48820)
0928 23:17:26.865304 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.867390 (+ 2086us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 23:17:26.867419 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:17:26.867460 6463 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 -> Peer cff6d43cf03a4cf7b839ed277bd30c33 (127.0.98.60:37281): Couldn't send request to peer cff6d43cf03a4cf7b839ed277bd30c33. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37281: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 23:17:26.870337 7136 master.cc:552] Master@127.0.98.60:37281 shutdown complete.
I20260928 23:17:26.870383 7136 master_runner.cc:305] Clearing existing system tablet
I20260928 23:17:26.870500 7136 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:26.870543 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:26.871182 7136 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:26.871701 7234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.871840 7136 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:26.871896 7136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.871918 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal
uuid: "cff6d43cf03a4cf7b839ed277bd30c33"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "1be3329fb4dfd931e25fa9275828aecd"
server_key_iv: "d11d0a27af18e9d2c3f9f6d471f04ebf"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.872622 7136 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:26.873878 7136 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 23:17:26.873940 7136 log.cc:1196] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 23:17:26.874105 7136 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: Deleting consensus metadata
I20260928 23:17:26.874331 7136 master_runner.cc:315] Copying system tablet from 127.0.98.62:41385
I20260928 23:17:26.874907 7136 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41385
I20260928 23:17:26.881577 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.875201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.0.98.62:41385)
0928 23:17:26.875246 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.875248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.875376 (+ 128us) server_negotiation.cc:407] Connection header received
0928 23:17:26.875423 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.875424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.875435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.875490 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.875842 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.876576 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.877548 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.877843 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.877905 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.878890 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.878892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.878892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.878900 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.880467 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.880836 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.880837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.880838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.880893 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.881079 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.881080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.881080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.881198 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.881259 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.881424 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.881496 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:17:26.881592 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.875146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:59598)
0928 23:17:26.875293 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.875302 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.875372 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.875494 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.875495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.875716 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.875772 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.875774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.876655 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.876656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.877460 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.877463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.877929 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.877930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.877987 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.878432 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.878437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.878803 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.880552 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.880553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.880553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.880764 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.880953 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.880954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.880954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.881019 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.881264 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.881265 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.881320 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.881397 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.881497 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260928 23:17:26.882104 6497 tablet_copy_service.cc:140] P c8e2b9ff9c38452fa792ffb0dfde9db9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cff6d43cf03a4cf7b839ed277bd30c33 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59598)
I20260928 23:17:26.882154 6497 tablet_copy_service.cc:161] P c8e2b9ff9c38452fa792ffb0dfde9db9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cff6d43cf03a4cf7b839ed277bd30c33 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59598: session id = cff6d43cf03a4cf7b839ed277bd30c33-00000000000000000000000000000000
I20260928 23:17:26.882670 6497 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:17:26.883117 7136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 23:17:26.884306 7136 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: tablet copy: Starting download of 0 data blocks...
I20260928 23:17:26.884426 7136 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: tablet copy: Starting download of 1 WAL segments...
I20260928 23:17:26.885190 7136 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:17:26.886063 6497 tablet_copy_service.cc:342] P c8e2b9ff9c38452fa792ffb0dfde9db9: Request end of tablet copy session cff6d43cf03a4cf7b839ed277bd30c33-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59598
I20260928 23:17:26.886111 6497 tablet_copy_service.cc:433] P c8e2b9ff9c38452fa792ffb0dfde9db9: ending tablet copy session cff6d43cf03a4cf7b839ed277bd30c33-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cff6d43cf03a4cf7b839ed277bd30c33
I20260928 23:17:26.888672 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:26.889786 7242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.889823 7241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.890211 7244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.890347 7136 server_base.cc:1034] running on GCE node
I20260928 23:17:26.890460 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:26.890614 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:26.891767 7136 hybrid_clock.cc:648] HybridClock initialized: now 1790637446891759 us; error 24 us; skew 500 ppm
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:26.893071 7136 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 23:17:26.893316 7136 webserver.cc:460] Webserver started at http://127.0.98.60:37051/ using document root <none> and password file <none>
I20260928 23:17:26.893421 7136 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:26.893452 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:26.894099 7136 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:26.894640 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.894773 7136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.894816 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal
uuid: "cff6d43cf03a4cf7b839ed277bd30c33"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "1be3329fb4dfd931e25fa9275828aecd"
server_key_iv: "d11d0a27af18e9d2c3f9f6d471f04ebf"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.894858 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:26.917948 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:26.919875 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:26.923084 7136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37281
I20260928 23:17:26.923125 7303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37281 every 8 connection(s)
I20260928 23:17:26.923571 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
I20260928 23:17:26.924536 7304 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:17:26.925036 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: Bootstrap starting.
I20260928 23:17:26.927925 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:17:26.928117 7304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: Bootstrap complete.
I20260928 23:17:26.928328 7304 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } }
I20260928 23:17:26.928417 7304 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cff6d43cf03a4cf7b839ed277bd30c33, State: Initialized, Role: LEARNER
I20260928 23:17:26.928462 7304 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } }
I20260928 23:17:26.928615 7304 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:26.928624 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:26.928833 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:17:26.929994 7318 catalog_manager.cc:1259] Loaded cluster ID: be543387447b4543ac1913c407ab356c
I20260928 23:17:26.930106 7318 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: loading cluster ID for follower catalog manager: success
I20260928 23:17:26.931361 7318 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: acquiring CA information for follower catalog manager: success
I20260928 23:17:26.931772 7318 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 23:17:26.950284 7229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:26.950475 7229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:26.950503 7229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:26.951610 7229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:26.951646 7229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:26.951680 7229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:26.953064 7229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36301
--tserver_master_addrs=127.0.98.62:41385,127.0.98.61:45471,127.0.98.60:37281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:26.953284 7229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:26.953485 7229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:26.958092 7326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.958127 7324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:26.958129 7323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:26.958333 7229 server_base.cc:1034] running on GCE node
I20260928 23:17:26.958508 7229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:26.958683 7229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:26.959817 7229 hybrid_clock.cc:648] HybridClock initialized: now 1790637446959804 us; error 27 us; skew 500 ppm
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:26.962041 7229 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 23:17:26.962428 7229 webserver.cc:460] Webserver started at http://127.0.98.1:36301/ using document root <none> and password file <none>
I20260928 23:17:26.962607 7229 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:26.962666 7229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:26.963933 7229 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:26.964493 7333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.964634 7229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.964710 7229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "0d3ab73582a544b993df86efe8831463"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "7f7e96ce80d3ea66a73f5aed0fc0111b"
server_key_iv: "4fa8a3105e2a1941637d1398f49a971a"
server_key_version: "encryptionkey@0"
I20260928 23:17:26.965018 7229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:26.979393 7229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:26.981141 7229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:26.981267 7229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:26.981464 7229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:26.981732 7229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:26.981763 7229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.981791 7229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:26.981814 7229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:26.987331 7229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44347
I20260928 23:17:26.987389 7446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44347 every 8 connection(s)
I20260928 23:17:26.987732 7229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 23:17:26 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 23:17:26.995937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 7229
I20260928 23:17:26.996083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 6673
I20260928 23:17:26.998194 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.988391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48171 (local address 127.0.98.61:45471)
0928 23:17:26.988440 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.988441 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.988647 (+ 206us) server_negotiation.cc:407] Connection header received
0928 23:17:26.989648 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.989650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.989662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.989691 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.990451 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.991181 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.992241 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.992506 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.992565 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.994820 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.994821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.994822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.994829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.996339 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.997262 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.997263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.997264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.997318 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.997527 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.997528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.997528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.997687 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.997741 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.998017 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.998092 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:17:26.998390 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.988717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37281 (local address 127.0.98.1:59051)
0928 23:17:26.988833 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.988843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.989552 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.989792 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.989793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.990182 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.990353 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.990358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.991493 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.991494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.992281 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.992282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.992686 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.992687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.992745 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.993108 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.993113 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.995023 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.997296 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.997297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.997297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.997481 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.997659 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.997660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.997660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.997709 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.997929 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.997930 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.997969 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.998028 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.998153 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":23,"thread_start_us":69,"threads_started":1}
I20260928 23:17:26.998392 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.988368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45471 (local address 127.0.98.1:48171)
0928 23:17:26.988518 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.988556 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.989532 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.989773 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.989777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.990176 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.990352 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.990358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.991263 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.991264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.992171 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.992173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.992569 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.992570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.992641 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.992970 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.992974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.994663 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:26.996983 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.996988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.996990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.997181 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.997373 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:26.997373 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:26.997374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:26.997475 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:26.997739 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:26.997746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:26.997876 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 23:17:26.997991 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 23:17:26.998209 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":99,"thread_start_us":83,"threads_started":1}
I20260928 23:17:26.998684 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.988786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59051 (local address 127.0.98.60:37281)
0928 23:17:26.988883 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.988885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.988893 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:26.989681 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.989682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.989694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.989751 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.990446 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.991418 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.992355 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.992575 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.992632 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.995110 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.995111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.995112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.995118 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:26.997213 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.997547 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.997548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.997548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.997601 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:26.997759 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:26.997760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:26.997761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:26.997867 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:26.997925 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:26.998522 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 23:17:26.998604 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":56,"threads_started":1}
I20260928 23:17:26.998821 7449 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45471
I20260928 23:17:26.998899 7449 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:26.999047 7449 heartbeater.cc:511] Master 127.0.98.61:45471 requested a full tablet report, sending...
I20260928 23:17:26.999493 7082 ts_manager.cc:194] Registered new tserver with Master: 0d3ab73582a544b993df86efe8831463 (127.0.98.1:44347)
I20260928 23:17:27.000871 7447 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37281
I20260928 23:17:27.000924 7447 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:27.000993 7447 heartbeater.cc:511] Master 127.0.98.60:37281 requested a full tablet report, sending...
I20260928 23:17:27.001327 7268 ts_manager.cc:194] Registered new tserver with Master: 0d3ab73582a544b993df86efe8831463 (127.0.98.1:44347)
I20260928 23:17:27.002252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33065
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41385
--webserver_port=34119
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41385,127.0.98.61:45471
--tserver_master_addrs=127.0.98.62:41385,127.0.98.61:45471,127.0.98.60:37281 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:17:27.002856 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.988369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.1:60195)
0928 23:17:26.988623 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:26.988632 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:26.989536 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:26.989777 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:26.989778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:26.990148 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:26.990352 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.990356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.991024 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.991024 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.991845 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:26.991847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.992153 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.992154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:26.992286 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.992783 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:26.992790 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:26.995022 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:27.002011 (+ 6989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.002011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.002012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.002160 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.002345 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.002346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.002346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.002406 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.002635 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:27.002636 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:27.002669 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.002715 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.002785 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260928 23:17:27.002882 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:26.988413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60195 (local address 127.0.98.62:41385)
0928 23:17:26.988483 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:26.988485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:26.988683 (+ 198us) server_negotiation.cc:407] Connection header received
0928 23:17:26.989628 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:26.989629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:26.989641 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:26.989704 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:26.990439 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.990934 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.991916 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:26.992091 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:26.992145 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:26.995089 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:26.995090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:26.995090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:26.995097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.001950 (+ 6853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.002225 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.002226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.002226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.002285 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.002461 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.002461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.002461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.002582 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.002642 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.002735 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.002802 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:17:27.003131 7448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41385
I20260928 23:17:27.003175 7448 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:27.003228 7448 heartbeater.cc:511] Master 127.0.98.62:41385 requested a full tablet report, sending...
I20260928 23:17:27.003449 6477 ts_manager.cc:194] Re-registered known tserver with Master: 0d3ab73582a544b993df86efe8831463 (127.0.98.1:44347)
I20260928 23:17:27.003811 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60195
W20260928 23:17:27.081405 7455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:27.081595 7455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:27.081616 7455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:27.082717 7455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:27.082758 7455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:27.082783 7455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:27.084206 7455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34119
--tserver_master_addrs=127.0.98.62:41385,127.0.98.61:45471,127.0.98.60:37281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:27.084420 7455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:27.084616 7455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:27.089424 7460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:27.089473 7455 server_base.cc:1034] running on GCE node
W20260928 23:17:27.089448 7461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:27.089425 7463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:27.089778 7455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:27.089972 7455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:27.091102 7455 hybrid_clock.cc:648] HybridClock initialized: now 1790637447091094 us; error 26 us; skew 500 ppm
Sep 28 23:17:27 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:27.093508 7455 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 23:17:27.093887 7455 webserver.cc:460] Webserver started at http://127.0.98.2:34119/ using document root <none> and password file <none>
I20260928 23:17:27.094089 7455 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:27.094141 7455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:27.095506 7455 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:27.096117 7470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:27.096297 7455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:27.096378 7455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "e507ca9b69c740e580572a7ae60c673b"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "01b46b6c3f455513a04da20c58f67b2e"
server_key_iv: "58a79b126723d56180ea7359c05833a3"
server_key_version: "encryptionkey@0"
I20260928 23:17:27.096714 7455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:27.112229 7455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:27.114234 7455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:27.114351 7455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:27.114547 7455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:27.114849 7455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:27.114888 7455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:27.114918 7455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:27.114941 7455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:27.120378 7455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33065
I20260928 23:17:27.120431 7583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33065 every 8 connection(s)
I20260928 23:17:27.120726 7455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:27.126839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 7455
I20260928 23:17:27.126983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 6807
Sep 28 23:17:27 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:17:27.131415 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.121589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35547 (local address 127.0.98.61:45471)
0928 23:17:27.121672 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.121674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.121852 (+ 178us) server_negotiation.cc:407] Connection header received
0928 23:17:27.122891 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.122892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.122900 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.122939 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.123608 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.124109 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.125490 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.125804 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.125870 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.128871 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.128873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.128873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.128880 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.130206 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.130545 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.130546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.130547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.130601 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.130842 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.130843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.130843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.130950 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.130981 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.131255 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.131327 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:17:27.131604 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.121341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45471 (local address 127.0.98.2:35547)
0928 23:17:27.121780 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.121789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.122619 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.122958 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.122959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.123267 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:27.123497 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.123502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.124233 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.124233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.125405 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.125407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.125881 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.125882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.126008 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.126505 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:27.126514 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:27.128770 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:27.130268 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.130274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.130277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.130483 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.130652 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.130653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.130653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.130766 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.131013 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:27.131018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:27.131130 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.131218 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.131365 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":51,"thread_start_us":81,"threads_started":1}
I20260928 23:17:27.131997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38679
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41385
--webserver_port=37105
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41385,127.0.98.61:45471
--tserver_master_addrs=127.0.98.62:41385,127.0.98.61:45471,127.0.98.60:37281 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:17:27.132045 7586 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45471
I20260928 23:17:27.132130 7586 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:27.132326 7586 heartbeater.cc:511] Master 127.0.98.61:45471 requested a full tablet report, sending...
I20260928 23:17:27.132763 7082 ts_manager.cc:194] Registered new tserver with Master: e507ca9b69c740e580572a7ae60c673b (127.0.98.2:33065)
Sep 28 23:17:27 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 23:17:27 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:27.137549 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.121327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59889 (local address 127.0.98.60:37281)
0928 23:17:27.121391 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.121393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.121603 (+ 210us) server_negotiation.cc:407] Connection header received
0928 23:17:27.122790 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.122792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.122805 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.122849 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.123628 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.124388 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.126123 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.126319 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.126374 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.134741 (+ 8367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.134743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.134743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.134750 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.136465 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.136853 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.136855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.136855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.136909 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.137089 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.137090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.137090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.137202 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.137255 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.137388 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.137465 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 23:17:27.137559 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.121341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37281 (local address 127.0.98.2:59889)
0928 23:17:27.121486 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.121520 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.122614 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.122868 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.122873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.123292 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:27.123499 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.123504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.125498 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.125498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.126054 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.126056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.126386 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.126387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.126468 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.126772 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:27.126775 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:27.134639 (+ 7864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:27.136543 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.136544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.136544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.136771 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.136972 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.136973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.136973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.137017 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.137258 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:27.137259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:27.137299 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.137361 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.137452 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":8,"thread_start_us":84,"threads_started":1}
I20260928 23:17:27.137625 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.121738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60623 (local address 127.0.98.62:41385)
0928 23:17:27.121783 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.121784 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.121912 (+ 128us) server_negotiation.cc:407] Connection header received
0928 23:17:27.122731 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.122733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.122745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.122790 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.123629 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.124394 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.125489 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.125804 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.125870 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.135071 (+ 9201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.135073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.135074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.135083 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.136402 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.136811 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.136813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.136813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.136874 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.137054 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.137055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.137055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.137184 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.137255 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.137451 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.137530 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:17:27.138155 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.121596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.2:60623)
0928 23:17:27.121876 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.121881 (+ 5us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.122629 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.122869 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.122873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.123304 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:27.123499 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.123504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.124462 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.124462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.125405 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.125407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.126219 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.126220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.126273 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.126614 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:27.126618 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:27.134957 (+ 8339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:27.136475 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.136476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.136476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.136738 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.136943 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.136944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.136944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.136998 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.137286 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:27.137287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:27.137335 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.137747 (+ 412us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.137852 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":26,"thread_start_us":51,"threads_started":1}
I20260928 23:17:27.138341 7584 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37281
I20260928 23:17:27.138388 7584 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:27.138453 7584 heartbeater.cc:511] Master 127.0.98.60:37281 requested a full tablet report, sending...
I20260928 23:17:27.138639 7585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41385
I20260928 23:17:27.138680 7585 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:27.138736 7585 heartbeater.cc:511] Master 127.0.98.62:41385 requested a full tablet report, sending...
I20260928 23:17:27.138765 7268 ts_manager.cc:194] Registered new tserver with Master: e507ca9b69c740e580572a7ae60c673b (127.0.98.2:33065)
I20260928 23:17:27.138942 6477 ts_manager.cc:194] Re-registered known tserver with Master: e507ca9b69c740e580572a7ae60c673b (127.0.98.2:33065)
I20260928 23:17:27.139294 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60623
W20260928 23:17:27.211762 7591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:27.211964 7591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:27.211985 7591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:27.213091 7591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:27.213126 7591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:27.213150 7591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:27.214531 7591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37105
--tserver_master_addrs=127.0.98.62:41385,127.0.98.61:45471,127.0.98.60:37281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:27.214730 7591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:27.214905 7591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:27.219707 7596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:27.219722 7597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:27.219841 7599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:27.219727 7591 server_base.cc:1034] running on GCE node
I20260928 23:17:27.220082 7591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:27.220321 7591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:27.221475 7591 hybrid_clock.cc:648] HybridClock initialized: now 1790637447221464 us; error 32 us; skew 500 ppm
Sep 28 23:17:27 dist-test-slave-djkr krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:27.223897 7591 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 23:17:27.224335 7591 webserver.cc:460] Webserver started at http://127.0.98.3:37105/ using document root <none> and password file <none>
I20260928 23:17:27.224540 7591 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:27.224601 7591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:27.225929 7591 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:27.226538 7606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:27.226716 7591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:27.226810 7591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "708099f2d6dd4974a8c2d8bd90b42666"
format_stamp: "Formatted at 2026-09-28 23:17:26 on dist-test-slave-djkr"
server_key: "e0c9a7ca36df0205ca47717b9454a401"
server_key_iv: "0d32571ad6367a0acdefb52ab0416cac"
server_key_version: "encryptionkey@0"
I20260928 23:17:27.227165 7591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:27.241348 7591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:27.243337 7591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:27.243461 7591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:27.243690 7591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:27.243969 7591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:27.244007 7591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:27.244035 7591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:27.244058 7591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:27.249747 7591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38679
I20260928 23:17:27.249814 7719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38679 every 8 connection(s)
I20260928 23:17:27.250108 7591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:17:27 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:27.256798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 7591
Sep 28 23:17:27 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 23:17:27 dist-test-slave-djkr krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:17:27.261516 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.250755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44393 (local address 127.0.98.60:37281)
0928 23:17:27.250798 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.250801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.251041 (+ 240us) server_negotiation.cc:407] Connection header received
0928 23:17:27.252114 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.252116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.252128 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.252208 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.253017 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.253510 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.254736 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.255015 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.255067 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.258441 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.258443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.258443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.258450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.260041 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.260497 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.260498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.260499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.260554 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.260817 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.260818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.260819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.260950 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.260990 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.261338 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.261415 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:17:27.261539 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.250752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36665 (local address 127.0.98.61:45471)
0928 23:17:27.250798 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.250799 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.251071 (+ 272us) server_negotiation.cc:407] Connection header received
0928 23:17:27.252219 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.252221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.252230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.252269 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.253045 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.253779 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.254788 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.255017 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.255077 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.258942 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.258943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.258943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.258952 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.260454 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.260773 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.260774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.260774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.260826 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.261012 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.261013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.261014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.261134 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.261177 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.261378 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.261451 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:17:27.262217 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.250760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37281 (local address 127.0.98.3:44393)
0928 23:17:27.250938 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.250959 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.252011 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.252340 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.252343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.252749 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:27.252914 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.252920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.253616 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.253617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.254650 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.254652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.255077 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.255078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.255140 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.255834 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:27.255839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:27.258366 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:27.260127 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.260133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.260136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.260422 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.260615 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.260615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.260616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.260744 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.261033 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:27.261039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:27.261199 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.261601 (+ 402us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.261994 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":23,"thread_start_us":99,"threads_started":1}
I20260928 23:17:27.262432 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.250760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45471 (local address 127.0.98.3:36665)
0928 23:17:27.250915 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.250954 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.252006 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.252339 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.252343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.252760 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:27.252914 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.252920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.253854 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.253854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.254730 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.254732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.255128 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.255129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.255196 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.255835 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:27.255839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:27.258630 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:27.260539 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.260540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.260540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.260713 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.260894 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.260895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.260895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.260948 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.261225 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:27.261225 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:27.261268 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.261802 (+ 534us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.262345 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":438,"thread_start_us":90,"threads_started":1}
I20260928 23:17:27.262787 7722 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45471
I20260928 23:17:27.262787 7720 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37281
I20260928 23:17:27.262899 7722 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:27.262899 7720 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:27.263106 7722 heartbeater.cc:511] Master 127.0.98.61:45471 requested a full tablet report, sending...
I20260928 23:17:27.263105 7720 heartbeater.cc:511] Master 127.0.98.60:37281 requested a full tablet report, sending...
I20260928 23:17:27.263526 7268 ts_manager.cc:194] Registered new tserver with Master: 708099f2d6dd4974a8c2d8bd90b42666 (127.0.98.3:38679)
I20260928 23:17:27.263535 7082 ts_manager.cc:194] Registered new tserver with Master: 708099f2d6dd4974a8c2d8bd90b42666 (127.0.98.3:38679)
I20260928 23:17:27.264268 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.251038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.3:36837)
0928 23:17:27.251208 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.251217 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.252019 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.252391 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.252392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.252723 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:27.252913 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.252918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.253880 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.253880 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.254547 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.254549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.254956 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.254957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.255090 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.255535 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:27.255566 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:27.260853 (+ 5287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:27.262848 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.262849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.262849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.263274 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.263482 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:27.263482 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:27.263482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:27.263565 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:27.263841 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:27.263842 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:27.263894 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.263947 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.264033 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":26,"thread_start_us":64,"threads_started":1}
I20260928 23:17:27.264554 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.251175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36837 (local address 127.0.98.62:41385)
0928 23:17:27.251227 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.251229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.251260 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:17:27.252318 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.252320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.252329 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.252368 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.252998 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.253772 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.254626 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.254886 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.254946 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.260937 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.260938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.260938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.260945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.262759 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.263359 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.263360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.263360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.263419 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.263618 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.263619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.263620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.263759 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.263817 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.264392 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.264466 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 23:17:27.265017 7721 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41385
I20260928 23:17:27.265313 7721 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:27.265386 7721 heartbeater.cc:511] Master 127.0.98.62:41385 requested a full tablet report, sending...
I20260928 23:17:27.265786 6477 ts_manager.cc:194] Re-registered known tserver with Master: 708099f2d6dd4974a8c2d8bd90b42666 (127.0.98.3:38679)
I20260928 23:17:27.266251 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36837
I20260928 23:17:27.267633 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.257893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.0.98.61:45471)
0928 23:17:27.257989 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.257991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.258098 (+ 107us) server_negotiation.cc:407] Connection header received
0928 23:17:27.258190 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.258192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.258203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.258267 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.259042 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.259820 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.261523 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.261842 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.261881 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.264848 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.264849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.264850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.264856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.266499 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.266883 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.266884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.266885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.266943 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.267125 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.267126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.267126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.267232 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.267358 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.267413 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.267489 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":51,"threads_started":1}
I20260928 23:17:27.268620 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.257777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.0.98.62:41385)
0928 23:17:27.257894 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.257895 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.258098 (+ 203us) server_negotiation.cc:407] Connection header received
0928 23:17:27.258195 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.258196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.258207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.258246 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.259041 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.259823 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.261958 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.262264 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.262330 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.265705 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.265707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.265707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.265713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.267474 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.267851 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.267852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.267852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.267904 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.268146 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.268147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.268148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.268254 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.268335 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.268454 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.268532 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20260928 23:17:27.270299 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.258024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.0.98.60:37281)
0928 23:17:27.258106 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.258108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.263774 (+ 5666us) server_negotiation.cc:407] Connection header received
0928 23:17:27.263796 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.263796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.263807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.263853 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.264426 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.265076 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.266163 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.266448 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.266505 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.267706 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.267707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.267708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.267715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.269304 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.269610 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.269612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.269612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.269667 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.269819 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.269820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.269820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.269955 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.270036 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.270135 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.270213 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"thread_start_us":44,"threads_started":1}
I20260928 23:17:27.269992 6477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:27.271510 6477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:27.278218 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.274340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38679 (local address 127.0.0.1:58230)
0928 23:17:27.274505 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.274515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.274549 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.274795 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.274796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.274798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:27.274837 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.274838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.276032 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.276033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.277194 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.277197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.278012 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.278013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.278068 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.278071 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.278100 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.278136 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":25}
I20260928 23:17:27.278214 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.274420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44347 (local address 127.0.0.1:41074)
0928 23:17:27.275035 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.275043 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.275087 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.275238 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.275239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.275241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:27.275277 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.275279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.276164 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.276165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.277121 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.277123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.277983 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.277984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.278041 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.278045 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.278090 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.278130 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":53,"threads_started":1}
I20260928 23:17:27.278417 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.274374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33065 (local address 127.0.0.1:45460)
0928 23:17:27.274916 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.274925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.274970 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.275335 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.275336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.275338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:27.275376 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.275378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.276538 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.276539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.277488 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.277490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.278220 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.278221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.278277 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.278281 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.278328 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.278361 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":79,"threads_started":1}
I20260928 23:17:27.278630 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.274692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.0.98.1:44347)
0928 23:17:27.274869 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.274873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.275088 (+ 215us) server_negotiation.cc:407] Connection header received
0928 23:17:27.275150 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.275152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.275165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.275229 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:27.275328 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.276076 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.277196 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.277899 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.277977 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.278476 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.278511 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.278557 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260928 23:17:27.278636 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.274521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.0.98.2:33065)
0928 23:17:27.274662 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.274666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.275136 (+ 470us) server_negotiation.cc:407] Connection header received
0928 23:17:27.275178 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.275180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.275208 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.275275 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:27.275436 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.276446 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.277554 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.278050 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.278106 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.278188 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.278517 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.278562 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260928 23:17:27.278784 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.274473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.98.3:38679)
0928 23:17:27.274625 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.274638 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.274648 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:27.274679 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.274680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.274691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.274773 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:27.274914 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.275943 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.277266 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.277927 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.278259 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.278548 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.278569 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.278607 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260928 23:17:27.279274 7381 tablet_service.cc:1467] Processing CreateTablet for tablet 201a4fb9aeb34b1096c2a674b056742c (DEFAULT_TABLE table=test-table [id=6c68c1ae0fac421494ec793348f7bc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:27.279428 7654 tablet_service.cc:1467] Processing CreateTablet for tablet 201a4fb9aeb34b1096c2a674b056742c (DEFAULT_TABLE table=test-table [id=6c68c1ae0fac421494ec793348f7bc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:27.279402 7518 tablet_service.cc:1467] Processing CreateTablet for tablet 201a4fb9aeb34b1096c2a674b056742c (DEFAULT_TABLE table=test-table [id=6c68c1ae0fac421494ec793348f7bc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:27.279644 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201a4fb9aeb34b1096c2a674b056742c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:27.279748 7654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201a4fb9aeb34b1096c2a674b056742c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:27.279748 7518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 201a4fb9aeb34b1096c2a674b056742c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:27.282295 7746 tablet_bootstrap.cc:492] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: Bootstrap starting.
I20260928 23:17:27.282809 7746 tablet_bootstrap.cc:654] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:27.283026 7747 tablet_bootstrap.cc:492] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: Bootstrap starting.
I20260928 23:17:27.283097 7746 log.cc:824] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:27.283653 7747 tablet_bootstrap.cc:654] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:27.283766 7748 tablet_bootstrap.cc:492] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: Bootstrap starting.
I20260928 23:17:27.283890 7746 tablet_bootstrap.cc:492] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: No bootstrap required, opened a new log
I20260928 23:17:27.283954 7746 ts_tablet_manager.cc:1397] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:27.284017 7747 log.cc:824] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:27.284341 7748 tablet_bootstrap.cc:654] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:27.284579 7748 log.cc:824] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:27.284705 7747 tablet_bootstrap.cc:492] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: No bootstrap required, opened a new log
I20260928 23:17:27.284770 7747 ts_tablet_manager.cc:1397] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:17:27.285352 7748 tablet_bootstrap.cc:492] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: No bootstrap required, opened a new log
I20260928 23:17:27.285418 7748 ts_tablet_manager.cc:1397] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:27.286307 7746 raft_consensus.cc:348] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } } peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } }
I20260928 23:17:27.286485 7746 raft_consensus.cc:374] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:27.286517 7746 raft_consensus.cc:729] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3ab73582a544b993df86efe8831463, State: Initialized, Role: FOLLOWER
I20260928 23:17:27.286672 7746 consensus_queue.cc:260] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } } peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } }
I20260928 23:17:27.286917 7746 ts_tablet_manager.cc:1428] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:17:27.286861 7748 raft_consensus.cc:348] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } } peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } }
I20260928 23:17:27.286984 7748 raft_consensus.cc:374] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:27.287027 7748 raft_consensus.cc:729] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e507ca9b69c740e580572a7ae60c673b, State: Initialized, Role: FOLLOWER
I20260928 23:17:27.287139 7748 consensus_queue.cc:260] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } } peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } }
I20260928 23:17:27.287233 7448 heartbeater.cc:503] Master 127.0.98.62:41385 was elected leader, sending a full tablet report...
I20260928 23:17:27.287783 7748 ts_tablet_manager.cc:1428] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:17:27.287725 7747 raft_consensus.cc:348] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } } peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } }
I20260928 23:17:27.287865 7747 raft_consensus.cc:374] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:27.287899 7747 raft_consensus.cc:729] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708099f2d6dd4974a8c2d8bd90b42666, State: Initialized, Role: FOLLOWER
I20260928 23:17:27.287994 7747 consensus_queue.cc:260] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } } peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } }
I20260928 23:17:27.288237 7747 ts_tablet_manager.cc:1428] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 23:17:27.288266 7721 heartbeater.cc:503] Master 127.0.98.62:41385 was elected leader, sending a full tablet report...
I20260928 23:17:27.289155 7585 heartbeater.cc:503] Master 127.0.98.62:41385 was elected leader, sending a full tablet report...
I20260928 23:17:27.294039 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.291749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.0.98.60:37281)
0928 23:17:27.291783 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.291785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.291794 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:27.291818 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.291818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.291837 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.291895 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:27.291976 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.292465 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.293300 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.293689 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.293751 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.293815 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.293932 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.293972 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:17:27.294040 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.291687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37281 (local address 127.0.0.1:48834)
0928 23:17:27.291734 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.291745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.291783 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.291899 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.291900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.291901 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:27.291940 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.291942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.292520 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.292520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.293227 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.293230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.293769 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.293791 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.293854 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.293876 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.293922 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.293973 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:17:27.294389 7278 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 23:17:27.295188 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e2b9ff9c38452fa792ffb0dfde9db9. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:27.295277 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:17:27.333432 7754 raft_consensus.cc:482] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:27.333567 7754 raft_consensus.cc:504] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } } peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } }
I20260928 23:17:27.333967 7754 leader_election.cc:288] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e507ca9b69c740e580572a7ae60c673b (127.0.98.2:33065), 0d3ab73582a544b993df86efe8831463 (127.0.98.1:44347)
I20260928 23:17:27.337749 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.334284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57679 (local address 127.0.98.2:33065)
0928 23:17:27.334484 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.334486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.334492 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:17:27.334510 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.334511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.334519 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.334560 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:27.334676 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.335357 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.336833 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.337381 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.337454 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.337464 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.337639 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.337678 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260928 23:17:27.337749 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.334273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49107 (local address 127.0.98.1:44347)
0928 23:17:27.334345 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.334347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.334356 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:27.334379 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.334380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.334389 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.334435 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:27.334582 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.335339 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.336830 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.337393 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.337454 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.337464 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.337639 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.337677 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:17:27.337770 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.334137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44347 (local address 127.0.98.3:49107)
0928 23:17:27.334225 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.334234 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.334265 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.334448 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.334449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.334450 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:27.334510 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.334512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.335439 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.335440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.336720 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.336723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.337470 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.337472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.337560 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.337565 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.337622 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.337671 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":37}
I20260928 23:17:27.337770 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.334157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33065 (local address 127.0.98.3:57679)
0928 23:17:27.334193 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:27.334205 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:27.334247 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:27.334578 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:27.334578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:27.334580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:27.334618 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.334619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.335439 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.335440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.336723 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:27.336724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.337470 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.337472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:27.337560 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.337565 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:17:27.337622 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:17:27.337671 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":55}
I20260928 23:17:27.338327 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a4fb9aeb34b1096c2a674b056742c" candidate_uuid: "708099f2d6dd4974a8c2d8bd90b42666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e507ca9b69c740e580572a7ae60c673b" is_pre_election: true
I20260928 23:17:27.338506 7538 raft_consensus.cc:2454] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 708099f2d6dd4974a8c2d8bd90b42666 in term 0.
I20260928 23:17:27.338565 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a4fb9aeb34b1096c2a674b056742c" candidate_uuid: "708099f2d6dd4974a8c2d8bd90b42666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3ab73582a544b993df86efe8831463" is_pre_election: true
I20260928 23:17:27.338713 7401 raft_consensus.cc:2454] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 708099f2d6dd4974a8c2d8bd90b42666 in term 0.
I20260928 23:17:27.338749 7610 leader_election.cc:302] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 708099f2d6dd4974a8c2d8bd90b42666, e507ca9b69c740e580572a7ae60c673b; no voters:
I20260928 23:17:27.338908 7754 raft_consensus.cc:2781] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:27.338972 7754 raft_consensus.cc:482] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:27.338994 7754 raft_consensus.cc:3037] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:27.339669 7754 raft_consensus.cc:504] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } } peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } }
I20260928 23:17:27.339780 7754 leader_election.cc:288] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [CANDIDATE]: Term 1 election: Requested vote from peers e507ca9b69c740e580572a7ae60c673b (127.0.98.2:33065), 0d3ab73582a544b993df86efe8831463 (127.0.98.1:44347)
I20260928 23:17:27.339922 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a4fb9aeb34b1096c2a674b056742c" candidate_uuid: "708099f2d6dd4974a8c2d8bd90b42666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e507ca9b69c740e580572a7ae60c673b"
I20260928 23:17:27.339938 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "201a4fb9aeb34b1096c2a674b056742c" candidate_uuid: "708099f2d6dd4974a8c2d8bd90b42666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3ab73582a544b993df86efe8831463"
I20260928 23:17:27.340003 7538 raft_consensus.cc:3037] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:27.340001 7401 raft_consensus.cc:3037] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:27.340816 7538 raft_consensus.cc:2454] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708099f2d6dd4974a8c2d8bd90b42666 in term 1.
I20260928 23:17:27.340816 7401 raft_consensus.cc:2454] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 708099f2d6dd4974a8c2d8bd90b42666 in term 1.
I20260928 23:17:27.340975 7608 leader_election.cc:302] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3ab73582a544b993df86efe8831463, 708099f2d6dd4974a8c2d8bd90b42666; no voters:
I20260928 23:17:27.341070 7754 raft_consensus.cc:2781] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:27.341229 7754 raft_consensus.cc:686] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 1 LEADER]: Becoming Leader. State: Replica: 708099f2d6dd4974a8c2d8bd90b42666, State: Running, Role: LEADER
I20260928 23:17:27.341317 7754 consensus_queue.cc:237] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } } peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } }
I20260928 23:17:27.342259 6477 catalog_manager.cc:5495] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 708099f2d6dd4974a8c2d8bd90b42666 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "708099f2d6dd4974a8c2d8bd90b42666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } health_report { overall_health: UNKNOWN } } }
I20260928 23:17:27.342727 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: true } } }
I20260928 23:17:27.342818 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:27.343349 7045 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9: attempting to promote NON_VOTER cff6d43cf03a4cf7b839ed277bd30c33 to VOTER
I20260928 23:17:27.343477 7045 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } }
I20260928 23:17:27.343885 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 FOLLOWER]: Refusing update from remote peer c8e2b9ff9c38452fa792ffb0dfde9db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 23:17:27.343935 7278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 LEARNER]: Refusing update from remote peer c8e2b9ff9c38452fa792ffb0dfde9db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 23:17:27.344058 7036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 23:17:27.344151 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 23:17:27.344594 7036 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, cff6d43cf03a4cf7b839ed277bd30c33 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:27.344698 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, cff6d43cf03a4cf7b839ed277bd30c33 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:27.345332 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:27.345404 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:27.345376 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:27.345449 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:27.345507 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:27.345584 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:17:27.345818 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:27.345888 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:27.345925 7278 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, cff6d43cf03a4cf7b839ed277bd30c33 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:27.346822 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:27.346903 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:17:27.349675 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.347576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.0.98.3:38679)
0928 23:17:27.347612 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.347614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.347681 (+ 67us) server_negotiation.cc:407] Connection header received
0928 23:17:27.347745 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.347745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.347760 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.347816 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:27.347912 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.348349 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.349071 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.349271 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.349333 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.349449 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:27.349543 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:27.349596 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.349611 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.349627 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:17:27.351258 7538 raft_consensus.cc:1261] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b [term 1 FOLLOWER]: Refusing update from remote peer 708099f2d6dd4974a8c2d8bd90b42666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:27.351279 7401 raft_consensus.cc:1261] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463 [term 1 FOLLOWER]: Refusing update from remote peer 708099f2d6dd4974a8c2d8bd90b42666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:27.351598 7759 consensus_queue.cc:1035] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:27.351720 7754 consensus_queue.cc:1035] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:27.353740 7763 mvcc.cc:205] Tried to move back new op lower bound from 7334450984348418048 to 7334450984310685696. Current Snapshot: MvccSnapshot[applied={T|T < 7334450984348418048}]
I20260928 23:17:27.354454 7764 mvcc.cc:205] Tried to move back new op lower bound from 7334450984348418048 to 7334450984310685696. Current Snapshot: MvccSnapshot[applied={T|T < 7334450984348418048}]
I20260928 23:17:27.358174 6477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:27.358251 6477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:27.359478 6477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8e2b9ff9c38452fa792ffb0dfde9db9: Sending DeleteTablet for 3 replicas of tablet 201a4fb9aeb34b1096c2a674b056742c
I20260928 23:17:27.359799 7654 tablet_service.cc:1514] Processing DeleteTablet for tablet 201a4fb9aeb34b1096c2a674b056742c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58230
I20260928 23:17:27.359894 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet 201a4fb9aeb34b1096c2a674b056742c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41074
I20260928 23:17:27.360069 7773 tablet_replica.cc:330] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: stopping tablet replica
I20260928 23:17:27.360190 7773 raft_consensus.cc:2229] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:27.360283 7774 tablet_replica.cc:330] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: stopping tablet replica
I20260928 23:17:27.360384 7774 raft_consensus.cc:2229] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:27.360389 7773 raft_consensus.cc:2258] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:27.360433 7518 tablet_service.cc:1514] Processing DeleteTablet for tablet 201a4fb9aeb34b1096c2a674b056742c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45460
I20260928 23:17:27.360494 7774 raft_consensus.cc:2258] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:27.360929 7774 ts_tablet_manager.cc:1905] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:27.360957 7773 ts_tablet_manager.cc:1905] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:27.361199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 6445
I20260928 23:17:27.363600 7774 ts_tablet_manager.cc:1918] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:27.363670 7774 log.cc:1196] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/201a4fb9aeb34b1096c2a674b056742c
I20260928 23:17:27.363632 7773 ts_tablet_manager.cc:1918] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:27.363862 7773 log.cc:1196] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/201a4fb9aeb34b1096c2a674b056742c
I20260928 23:17:27.363917 7774 ts_tablet_manager.cc:1939] T 201a4fb9aeb34b1096c2a674b056742c P 0d3ab73582a544b993df86efe8831463: Deleting consensus metadata
I20260928 23:17:27.364238 7773 ts_tablet_manager.cc:1939] T 201a4fb9aeb34b1096c2a674b056742c P 708099f2d6dd4974a8c2d8bd90b42666: Deleting consensus metadata
W20260928 23:17:27.366413 7335 connection.cc:535] server connection from 127.0.0.1:41074 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 23:17:27.366420 7608 connection.cc:535] server connection from 127.0.0.1:58230 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:17:27.367146 7776 tablet_replica.cc:330] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: stopping tablet replica
I20260928 23:17:27.367269 7776 raft_consensus.cc:2229] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:27.367378 7776 raft_consensus.cc:2258] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:27.368146 7776 ts_tablet_manager.cc:1905] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:27.369721 7776 ts_tablet_manager.cc:1918] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:27.369781 7776 log.cc:1196] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/201a4fb9aeb34b1096c2a674b056742c
I20260928 23:17:27.370007 7776 ts_tablet_manager.cc:1939] T 201a4fb9aeb34b1096c2a674b056742c P e507ca9b69c740e580572a7ae60c673b: Deleting consensus metadata
W20260928 23:17:27.370431 7472 connection.cc:441] server connection from 127.0.0.1:45460 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45460 (request call id 1) could send its response
I20260928 23:17:27.375134 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.360716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.0.98.60:37281)
0928 23:17:27.360759 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.360761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.367605 (+ 6844us) server_negotiation.cc:407] Connection header received
0928 23:17:27.367714 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.367715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.367726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.367755 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.368588 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.369357 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.370416 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.370692 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.370729 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.372501 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.372502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.372503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.372509 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.373976 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.374323 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.374325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.374325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.374380 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.374576 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.374577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.374577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.374701 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.374793 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.374936 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.375038 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:17:27.377143 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.368328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.0.98.60:37281)
0928 23:17:27.368367 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.368369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.368864 (+ 495us) server_negotiation.cc:407] Connection header received
0928 23:17:27.368947 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.368948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.368957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.368983 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.369591 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.370295 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.371202 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.371516 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.371617 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.373800 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.373801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.373802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.373808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.375469 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.376278 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.376279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.376279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.376334 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.376506 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.376507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.376507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.376634 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.376929 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.376971 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.377035 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:17:27.378386 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:27.368618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.0.98.61:45471)
0928 23:17:27.368661 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:27.368663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:27.371336 (+ 2673us) server_negotiation.cc:407] Connection header received
0928 23:17:27.371426 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:27.371427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:27.371440 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:27.371469 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:27.372117 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.372865 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.374295 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:27.374751 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:27.374792 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:27.376019 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:27.376020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:27.376020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:27.376027 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:27.377449 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.377718 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.377719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.377719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.377779 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:27.377937 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:27.377938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:27.377938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:27.378041 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:27.378121 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:27.378226 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 23:17:27.378301 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:17:28.356794 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.356492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.3:60415)
0928 23:17:28.356678 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.356752 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
I20260928 23:17:28.356948 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.356738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.1:46231)
0928 23:17:28.356835 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.356904 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":45,"threads_started":1}
I20260928 23:17:28.356997 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.356522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.2:52041)
0928 23:17:28.356902 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.356961 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":48,"threads_started":1}
W20260928 23:17:28.356994 7721 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
W20260928 23:17:28.357158 7585 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
W20260928 23:17:28.357187 7448 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41385 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
I20260928 23:17:28.859819 7787 raft_consensus.cc:482] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8e2b9ff9c38452fa792ffb0dfde9db9)
I20260928 23:17:28.859905 7787 raft_consensus.cc:504] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } }
I20260928 23:17:28.860047 7787 leader_election.cc:288] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8e2b9ff9c38452fa792ffb0dfde9db9 (127.0.98.62:41385), cff6d43cf03a4cf7b839ed277bd30c33 (127.0.98.60:37281)
I20260928 23:17:28.860126 7788 raft_consensus.cc:482] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8e2b9ff9c38452fa792ffb0dfde9db9)
I20260928 23:17:28.860250 7788 raft_consensus.cc:504] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } }
I20260928 23:17:28.860507 7788 leader_election.cc:288] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8e2b9ff9c38452fa792ffb0dfde9db9 (127.0.98.62:41385), a61681d380f941c794e06efb97fffac5 (127.0.98.61:45471)
I20260928 23:17:28.860720 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.860562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:60082)
0928 23:17:28.860635 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.860683 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56,"thread_start_us":36,"threads_started":1}
I20260928 23:17:28.860728 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.860174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:60074)
0928 23:17:28.860492 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.860687 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":46,"threads_started":1}
W20260928 23:17:28.860873 7254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
W20260928 23:17:28.860882 7068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
I20260928 23:17:28.861248 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.861165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:60096)
0928 23:17:28.861201 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.861223 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:17:28.861264 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.861161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:60086)
0928 23:17:28.861201 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.861235 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:17:28.861363 7068 leader_election.cc:334] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8e2b9ff9c38452fa792ffb0dfde9db9 (127.0.98.62:41385): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
W20260928 23:17:28.861397 7254 leader_election.cc:334] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c8e2b9ff9c38452fa792ffb0dfde9db9 (127.0.98.62:41385): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
I20260928 23:17:28.863480 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.860423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.0.98.60:37281)
0928 23:17:28.860714 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:28.860715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:28.860722 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:28.860746 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:28.860746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:28.860755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:28.861228 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:28.861263 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.861832 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.862799 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.863218 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.863269 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.863278 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:28.863407 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 23:17:28.863432 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":53,"threads_started":1}
I20260928 23:17:28.863507 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.860314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37281 (local address 127.0.0.1:48862)
0928 23:17:28.860410 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.860422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:28.860481 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:28.861103 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:28.861104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:28.861107 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:28.861168 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.861170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.861906 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.861907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.862707 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.862709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.863279 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.863280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.863343 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.863347 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:28.863400 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:17:28.863439 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":1,"thread_start_us":33,"threads_started":1}
I20260928 23:17:28.863785 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.860627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.0.98.61:45471)
0928 23:17:28.860705 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:28.860706 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:28.861042 (+ 336us) server_negotiation.cc:407] Connection header received
0928 23:17:28.861070 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:28.861071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:28.861081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:28.861481 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:28.861512 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.862028 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.863029 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.863438 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.863492 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.863500 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:28.863681 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 23:17:28.863716 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62,"thread_start_us":42,"threads_started":1}
I20260928 23:17:28.863720 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a61681d380f941c794e06efb97fffac5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" is_pre_election: true
I20260928 23:17:28.863806 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.860582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45471 (local address 127.0.0.1:60622)
0928 23:17:28.860780 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.860788 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:28.860871 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:28.861284 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:28.861284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:28.861286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:28.861334 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.861336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.862122 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.862123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.862952 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.862953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.863502 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.863503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.863587 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.863591 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:28.863644 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:17:28.863682 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":40,"threads_started":1}
I20260928 23:17:28.863842 7278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a61681d380f941c794e06efb97fffac5 in term 1.
I20260928 23:17:28.863957 7068 leader_election.cc:302] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a61681d380f941c794e06efb97fffac5, cff6d43cf03a4cf7b839ed277bd30c33; no voters: c8e2b9ff9c38452fa792ffb0dfde9db9
I20260928 23:17:28.864037 7787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 23:17:28.864077 7787 raft_consensus.cc:482] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c8e2b9ff9c38452fa792ffb0dfde9db9)
I20260928 23:17:28.864097 7787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:17:28.864111 7092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a61681d380f941c794e06efb97fffac5" is_pre_election: true
I20260928 23:17:28.864770 7787 raft_consensus.cc:504] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } }
I20260928 23:17:28.864883 7787 leader_election.cc:288] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [CANDIDATE]: Term 2 election: Requested vote from peers c8e2b9ff9c38452fa792ffb0dfde9db9 (127.0.98.62:41385), cff6d43cf03a4cf7b839ed277bd30c33 (127.0.98.60:37281)
I20260928 23:17:28.864895 7092 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cff6d43cf03a4cf7b839ed277bd30c33 in current term 2: Already voted for candidate a61681d380f941c794e06efb97fffac5 in this term.
I20260928 23:17:28.865089 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.864999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:60110)
0928 23:17:28.865040 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.865067 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 23:17:28.865176 7253 leader_election.cc:302] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cff6d43cf03a4cf7b839ed277bd30c33; no voters: a61681d380f941c794e06efb97fffac5, c8e2b9ff9c38452fa792ffb0dfde9db9
I20260928 23:17:28.865187 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a61681d380f941c794e06efb97fffac5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cff6d43cf03a4cf7b839ed277bd30c33"
I20260928 23:17:28.865361 7278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:17:28.865465 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.865375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:60126)
0928 23:17:28.865419 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.865444 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:17:28.865523 7068 leader_election.cc:334] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c8e2b9ff9c38452fa792ffb0dfde9db9 (127.0.98.62:41385): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
I20260928 23:17:28.865967 7278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a61681d380f941c794e06efb97fffac5 in term 2.
I20260928 23:17:28.866034 7788 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:17:28.866119 7068 leader_election.cc:302] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a61681d380f941c794e06efb97fffac5, cff6d43cf03a4cf7b839ed277bd30c33; no voters: c8e2b9ff9c38452fa792ffb0dfde9db9
I20260928 23:17:28.866196 7787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:17:28.866320 7787 raft_consensus.cc:686] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [term 2 LEADER]: Becoming Leader. State: Replica: a61681d380f941c794e06efb97fffac5, State: Running, Role: LEADER
I20260928 23:17:28.866381 7787 consensus_queue.cc:237] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } }
I20260928 23:17:28.866680 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a61681d380f941c794e06efb97fffac5. Latest consensus state: current_term: 2 leader_uuid: "a61681d380f941c794e06efb97fffac5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:28.866770 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:28.866933 7799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:28.867331 7799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:28.867391 7799 catalog_manager.cc:1259] Loaded cluster ID: be543387447b4543ac1913c407ab356c
I20260928 23:17:28.867420 7799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:28.867616 7799 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:28.867688 7799 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Loaded TSK: 0
I20260928 23:17:28.867849 7799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:17:28.920159 7082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:28.921725 7082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:28.922767 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.922616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:60144)
0928 23:17:28.922692 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.922734 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 23:17:28.922873 7278 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [term 2 FOLLOWER]: Refusing update from remote peer a61681d380f941c794e06efb97fffac5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 23:17:28.923156 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.923062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.0.1:60152)
0928 23:17:28.923106 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.923130 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:17:28.923161 7787 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 23:17:28.923336 7068 consensus_peers.cc:487] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 -> Peer c8e2b9ff9c38452fa792ffb0dfde9db9 (127.0.98.62:41385): Couldn't send request to peer c8e2b9ff9c38452fa792ffb0dfde9db9. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 23:17:28.923875 7788 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a61681d380f941c794e06efb97fffac5. Latest consensus state: current_term: 2 leader_uuid: "a61681d380f941c794e06efb97fffac5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:28.923964 7788 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:17:28.924249 7800 mvcc.cc:205] Tried to move back new op lower bound from 7334450990785597440 to 7334450990557188096. Current Snapshot: MvccSnapshot[applied={T|T < 7334450990785597440}]
I20260928 23:17:28.924352 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a61681d380f941c794e06efb97fffac5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:28.924441 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:28.924543 7788 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a61681d380f941c794e06efb97fffac5" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c8e2b9ff9c38452fa792ffb0dfde9db9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41385 } } peers { permanent_uuid: "a61681d380f941c794e06efb97fffac5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45471 } attrs { promote: false } } peers { permanent_uuid: "cff6d43cf03a4cf7b839ed277bd30c33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37281 } attrs { promote: false } } }
I20260928 23:17:28.924611 7788 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff6d43cf03a4cf7b839ed277bd30c33 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:17:28.929330 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.926317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.0.98.3:38679)
0928 23:17:28.926500 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:28.926501 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:28.926508 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:28.926530 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:28.926531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:28.926541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:28.926595 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:28.926746 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.927357 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.928573 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.928979 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.929042 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.929054 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:28.929194 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:17:28.929231 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":45,"threads_started":1}
I20260928 23:17:28.929311 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.925998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38679 (local address 127.0.0.1:58246)
0928 23:17:28.926118 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.926125 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:28.926162 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:28.926635 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:28.926636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:28.926638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:28.926684 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.926686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.927420 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.927421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.928500 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.928502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.929075 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.929076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.929139 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.929143 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:28.929190 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:17:28.929231 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260928 23:17:28.929437 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.925954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33065 (local address 127.0.0.1:45476)
0928 23:17:28.926105 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.926115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:28.926153 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:28.926549 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:28.926550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:28.926551 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:28.926581 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.926583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.927177 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.927178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.928290 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.928292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.929241 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.929242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.929298 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.929302 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:28.929326 (+ 24us) client_negotiation.cc:240] Negotiation successful
0928 23:17:28.929370 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260928 23:17:28.929631 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.926205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.0.98.2:33065)
0928 23:17:28.926425 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:28.926427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:28.926435 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:28.926457 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:28.926458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:28.926467 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:28.926636 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:28.926669 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.927110 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.928583 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.929152 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.929500 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.929513 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:28.929531 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 23:17:28.929568 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":62,"threads_started":1}
I20260928 23:17:28.929826 7518 tablet_service.cc:1467] Processing CreateTablet for tablet 63c47e0edacd4728848d98fe967d7f2e (DEFAULT_TABLE table=test-table [id=adff653f44f742048b3a1309030b2c3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:28.929960 7518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c47e0edacd4728848d98fe967d7f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:28.930049 7654 tablet_service.cc:1467] Processing CreateTablet for tablet 63c47e0edacd4728848d98fe967d7f2e (DEFAULT_TABLE table=test-table [id=adff653f44f742048b3a1309030b2c3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:28.930155 7654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c47e0edacd4728848d98fe967d7f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:28.930459 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.925911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44347 (local address 127.0.0.1:41080)
0928 23:17:28.925985 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.925995 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:28.926042 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:28.926258 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:28.926259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:28.926261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:28.926314 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.926315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.928382 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.928383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.929457 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.929459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.930220 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.930221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.930302 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.930305 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:28.930350 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:17:28.930387 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 23:17:28.930462 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.926063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.0.98.1:44347)
0928 23:17:28.926150 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:28.926152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:28.926159 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:28.926183 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:28.926184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:28.926193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:28.926247 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:28.926371 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.927101 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.929540 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.930097 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.930191 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.930234 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:28.930361 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 23:17:28.930396 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":48,"threads_started":1}
I20260928 23:17:28.931360 7809 tablet_bootstrap.cc:492] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b: Bootstrap starting.
I20260928 23:17:28.931375 7810 tablet_bootstrap.cc:492] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: Bootstrap starting.
I20260928 23:17:28.931699 7381 tablet_service.cc:1467] Processing CreateTablet for tablet 63c47e0edacd4728848d98fe967d7f2e (DEFAULT_TABLE table=test-table [id=adff653f44f742048b3a1309030b2c3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:28.931756 7810 tablet_bootstrap.cc:654] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:28.931754 7809 tablet_bootstrap.cc:654] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:28.931810 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c47e0edacd4728848d98fe967d7f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:28.932255 7810 tablet_bootstrap.cc:492] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: No bootstrap required, opened a new log
I20260928 23:17:28.932297 7810 ts_tablet_manager.cc:1397] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:28.932355 7809 tablet_bootstrap.cc:492] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b: No bootstrap required, opened a new log
I20260928 23:17:28.932380 7810 raft_consensus.cc:348] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } } peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } }
I20260928 23:17:28.932427 7810 raft_consensus.cc:374] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:28.932438 7810 raft_consensus.cc:729] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708099f2d6dd4974a8c2d8bd90b42666, State: Initialized, Role: FOLLOWER
I20260928 23:17:28.932471 7810 consensus_queue.cc:260] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } } peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } }
I20260928 23:17:28.932571 7809 ts_tablet_manager.cc:1397] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:28.932595 7810 ts_tablet_manager.cc:1428] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:28.932756 7809 raft_consensus.cc:348] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } } peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } }
I20260928 23:17:28.932878 7809 raft_consensus.cc:374] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:28.932934 7809 raft_consensus.cc:729] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e507ca9b69c740e580572a7ae60c673b, State: Initialized, Role: FOLLOWER
I20260928 23:17:28.933025 7809 consensus_queue.cc:260] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } } peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } }
I20260928 23:17:28.933284 7809 ts_tablet_manager.cc:1428] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:17:28.933257 7722 heartbeater.cc:503] Master 127.0.98.61:45471 was elected leader, sending a full tablet report...
I20260928 23:17:28.933321 7814 tablet_bootstrap.cc:492] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463: Bootstrap starting.
I20260928 23:17:28.933604 7814 tablet_bootstrap.cc:654] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:28.934192 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.933993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.2:51759)
0928 23:17:28.934117 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.934152 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260928 23:17:28.934199 7586 heartbeater.cc:503] Master 127.0.98.61:45471 was elected leader, sending a full tablet report...
I20260928 23:17:28.934378 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.933203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.3:54615)
0928 23:17:28.934227 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.934348 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1004,"thread_start_us":92,"threads_started":1}
I20260928 23:17:28.934752 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.934672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.3:54273)
0928 23:17:28.934710 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.934730 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:17:28.934830 7814 tablet_bootstrap.cc:492] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463: No bootstrap required, opened a new log
I20260928 23:17:28.934882 7814 ts_tablet_manager.cc:1397] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:28.935029 7814 raft_consensus.cc:348] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } } peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } }
I20260928 23:17:28.935079 7814 raft_consensus.cc:374] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:28.935096 7814 raft_consensus.cc:729] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3ab73582a544b993df86efe8831463, State: Initialized, Role: FOLLOWER
I20260928 23:17:28.935149 7814 consensus_queue.cc:260] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } } peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } }
I20260928 23:17:28.935294 7814 ts_tablet_manager.cc:1428] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:28.935632 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.935431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.1:41605)
0928 23:17:28.935562 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.935594 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
I20260928 23:17:28.935670 7449 heartbeater.cc:503] Master 127.0.98.61:45471 was elected leader, sending a full tablet report...
I20260928 23:17:28.936033 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.935949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.1:45743)
0928 23:17:28.935980 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.936008 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 23:17:28.989357 7450 tablet.cc:2369] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:28.990875 7815 raft_consensus.cc:482] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:28.990962 7815 raft_consensus.cc:504] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } } peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } }
I20260928 23:17:28.991326 7815 leader_election.cc:288] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d3ab73582a544b993df86efe8831463 (127.0.98.1:44347), 708099f2d6dd4974a8c2d8bd90b42666 (127.0.98.3:38679)
I20260928 23:17:28.993916 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.991570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58255 (local address 127.0.98.1:44347)
0928 23:17:28.991610 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:28.991612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:28.991622 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:28.991689 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:28.991690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:28.991700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:28.991755 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:28.991886 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.992357 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.993281 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.993619 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.993676 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.993688 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:28.993825 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 23:17:28.993855 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:17:28.993919 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.991473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44347 (local address 127.0.98.2:58255)
0928 23:17:28.991533 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.991564 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:28.991630 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:28.991757 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:28.991758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:28.991760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:28.991825 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.991827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.992425 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.992426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.993199 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.993201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.993691 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.993692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.993764 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.993768 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:28.993810 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:17:28.993848 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":26}
I20260928 23:17:28.993973 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.991646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51023 (local address 127.0.98.3:38679)
0928 23:17:28.991687 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:28.991689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:28.991802 (+ 113us) server_negotiation.cc:407] Connection header received
0928 23:17:28.991861 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:28.991862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:28.991874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:28.991919 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:28.992044 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.992504 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.993345 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.993688 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.993742 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.993752 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:28.993880 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 23:17:28.993912 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:17:28.994163 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.991594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38679 (local address 127.0.98.2:51023)
0928 23:17:28.991749 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.991755 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:28.991794 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:28.991928 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:28.991929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:28.991930 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:28.991990 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.991991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.992570 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.992571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.993267 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:28.993269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:28.993753 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:28.993754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:28.993819 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:28.993822 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:28.993866 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:17:28.993904 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260928 23:17:28.994314 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c47e0edacd4728848d98fe967d7f2e" candidate_uuid: "e507ca9b69c740e580572a7ae60c673b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3ab73582a544b993df86efe8831463" is_pre_election: true
I20260928 23:17:28.994410 7401 raft_consensus.cc:2454] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e507ca9b69c740e580572a7ae60c673b in term 0.
I20260928 23:17:28.994403 7674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c47e0edacd4728848d98fe967d7f2e" candidate_uuid: "e507ca9b69c740e580572a7ae60c673b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "708099f2d6dd4974a8c2d8bd90b42666" is_pre_election: true
I20260928 23:17:28.994529 7674 raft_consensus.cc:2454] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e507ca9b69c740e580572a7ae60c673b in term 0.
I20260928 23:17:28.994688 7472 leader_election.cc:302] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3ab73582a544b993df86efe8831463, e507ca9b69c740e580572a7ae60c673b; no voters:
I20260928 23:17:28.994818 7815 raft_consensus.cc:2781] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:28.994866 7815 raft_consensus.cc:482] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:28.994884 7815 raft_consensus.cc:3037] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:28.995587 7815 raft_consensus.cc:504] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } } peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } }
I20260928 23:17:28.995697 7815 leader_election.cc:288] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [CANDIDATE]: Term 1 election: Requested vote from peers 0d3ab73582a544b993df86efe8831463 (127.0.98.1:44347), 708099f2d6dd4974a8c2d8bd90b42666 (127.0.98.3:38679)
I20260928 23:17:28.995836 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c47e0edacd4728848d98fe967d7f2e" candidate_uuid: "e507ca9b69c740e580572a7ae60c673b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3ab73582a544b993df86efe8831463"
I20260928 23:17:28.995872 7674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c47e0edacd4728848d98fe967d7f2e" candidate_uuid: "e507ca9b69c740e580572a7ae60c673b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "708099f2d6dd4974a8c2d8bd90b42666"
I20260928 23:17:28.995911 7401 raft_consensus.cc:3037] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:28.995934 7674 raft_consensus.cc:3037] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:28.996518 7401 raft_consensus.cc:2454] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e507ca9b69c740e580572a7ae60c673b in term 1.
I20260928 23:17:28.996522 7674 raft_consensus.cc:2454] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e507ca9b69c740e580572a7ae60c673b in term 1.
I20260928 23:17:28.996682 7472 leader_election.cc:302] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3ab73582a544b993df86efe8831463, e507ca9b69c740e580572a7ae60c673b; no voters:
I20260928 23:17:28.996752 7815 raft_consensus.cc:2781] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:28.996897 7815 raft_consensus.cc:686] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 1 LEADER]: Becoming Leader. State: Replica: e507ca9b69c740e580572a7ae60c673b, State: Running, Role: LEADER
I20260928 23:17:28.997059 7815 consensus_queue.cc:237] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } } peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } }
I20260928 23:17:28.997166 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:28.997075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.2:57231)
0928 23:17:28.997117 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:28.997142 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 23:17:28.997732 7081 catalog_manager.cc:5495] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b reported cstate change: term changed from 0 to 1, leader changed from <none> to e507ca9b69c740e580572a7ae60c673b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e507ca9b69c740e580572a7ae60c673b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e507ca9b69c740e580572a7ae60c673b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33065 } health_report { overall_health: HEALTHY } } }
W20260928 23:17:29.001646 7723 tablet.cc:2369] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:29.019785 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:29.017622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.0.98.2:33065)
0928 23:17:29.017678 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:29.017680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:29.017688 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:29.017723 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:29.017724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:29.017743 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:29.017801 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:29.017913 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:29.018386 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:29.019081 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:29.019266 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:29.019317 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:29.019459 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:29.019588 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:29.019648 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:29.019696 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:17:29.019715 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:17:29.020835 7401 raft_consensus.cc:1261] T 63c47e0edacd4728848d98fe967d7f2e P 0d3ab73582a544b993df86efe8831463 [term 1 FOLLOWER]: Refusing update from remote peer e507ca9b69c740e580572a7ae60c673b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:29.020896 7674 raft_consensus.cc:1261] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666 [term 1 FOLLOWER]: Refusing update from remote peer e507ca9b69c740e580572a7ae60c673b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:29.021097 7815 consensus_queue.cc:1035] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3ab73582a544b993df86efe8831463" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:29.021229 7822 consensus_queue.cc:1035] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [LEADER]: Connected to new peer: Peer: permanent_uuid: "708099f2d6dd4974a8c2d8bd90b42666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:29.022506 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:29.022409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.1:45907)
0928 23:17:29.022448 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:29.022476 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:17:29.022959 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:29.022672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.3:51571)
0928 23:17:29.022761 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:29.022796 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 23:17:29.023859 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:29.023772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.2:39919)
0928 23:17:29.023804 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:29.023831 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 23:17:29.023902 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:29.023797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.3:40595)
0928 23:17:29.023857 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:29.023880 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 23:17:29.024183 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:29.024100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.2:46337)
0928 23:17:29.024136 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:29.024156 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:17:29.024224 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:29.024139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41385 (local address 127.0.98.1:35247)
0928 23:17:29.024175 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:29.024202 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 23:17:29.024523 7827 mvcc.cc:205] Tried to move back new op lower bound from 7334450991187369984 to 7334450991092486144. Current Snapshot: MvccSnapshot[applied={T|T < 7334450991187369984}]
I20260928 23:17:29.026403 7081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:29.026494 7081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:29.027667 7081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61681d380f941c794e06efb97fffac5: Sending DeleteTablet for 3 replicas of tablet 63c47e0edacd4728848d98fe967d7f2e
I20260928 23:17:29.027931 7518 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c47e0edacd4728848d98fe967d7f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:29 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:45476
I20260928 23:17:29.027998 7654 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c47e0edacd4728848d98fe967d7f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:29 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:58246
I20260928 23:17:29.027997 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c47e0edacd4728848d98fe967d7f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:29 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41080
I20260928 23:17:29.028298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 7229
I20260928 23:17:29.028329 7837 tablet_replica.cc:330] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: stopping tablet replica
I20260928 23:17:29.030498 7836 tablet_replica.cc:330] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b: stopping tablet replica
I20260928 23:17:29.030551 7836 raft_consensus.cc:2229] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:29.030633 7836 raft_consensus.cc:2258] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:29.030706 7837 raft_consensus.cc:2229] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:29.030762 7837 raft_consensus.cc:2258] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:29.031759 7836 ts_tablet_manager.cc:1905] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:29.033437 7836 ts_tablet_manager.cc:1918] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:29.033440 7837 ts_tablet_manager.cc:1905] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:29.033551 7836 log.cc:1196] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/63c47e0edacd4728848d98fe967d7f2e
I20260928 23:17:29.033886 7836 ts_tablet_manager.cc:1939] T 63c47e0edacd4728848d98fe967d7f2e P e507ca9b69c740e580572a7ae60c673b: Deleting consensus metadata
I20260928 23:17:29.034228 7069 catalog_manager.cc:4841] TS e507ca9b69c740e580572a7ae60c673b (127.0.98.2:33065): tablet 63c47e0edacd4728848d98fe967d7f2e (table test-table [id=adff653f44f742048b3a1309030b2c3c]) successfully deleted
W20260928 23:17:29.034495 7789 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:29.033923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44347 (local address 127.0.0.1:41086)
0928 23:17:29.033968 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:29.033978 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:29.034012 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:29.034450 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44347: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 23:17:29.034603 7067 catalog_manager.cc:4584] TS 0d3ab73582a544b993df86efe8831463 (127.0.98.1:44347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63c47e0edacd4728848d98fe967d7f2e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44347: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:17:29.034914 7837 ts_tablet_manager.cc:1918] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:29.034947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 7455
I20260928 23:17:29.034974 7837 log.cc:1196] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/63c47e0edacd4728848d98fe967d7f2e
I20260928 23:17:29.036034 7837 ts_tablet_manager.cc:1939] T 63c47e0edacd4728848d98fe967d7f2e P 708099f2d6dd4974a8c2d8bd90b42666: Deleting consensus metadata
I20260928 23:17:29.036355 7067 catalog_manager.cc:4841] TS 708099f2d6dd4974a8c2d8bd90b42666 (127.0.98.3:38679): tablet 63c47e0edacd4728848d98fe967d7f2e (table test-table [id=adff653f44f742048b3a1309030b2c3c]) successfully deleted
I20260928 23:17:29.039614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 7591
I20260928 23:17:29.044734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 6941
I20260928 23:17:29.048606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 7136
2026-09-28T23:17:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4299 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:29 dist-test-slave-djkr krb5kdc[7841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:29 dist-test-slave-djkr krb5kdc[7841](info): set up 2 sockets
Sep 28 23:17:29 dist-test-slave-djkr krb5kdc[7841](info): commencing operation
krb5kdc: starting...
W20260928 23:17:29.991642 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 23:17:30.101816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:30.178602 7854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:30.178792 7854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:30.178820 7854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:30.179972 7854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:30.180007 7854 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:17:30.180024 7854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:30.180040 7854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:30.180056 7854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:30.180080 7854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:30.181488 7854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45297
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:30.181711 7854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:30.181897 7854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:30.186388 7862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:30.186420 7860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:30.186394 7859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:30.186452 7854 server_base.cc:1034] running on GCE node
I20260928 23:17:30.186813 7854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:30.187089 7854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:30.188241 7854 hybrid_clock.cc:648] HybridClock initialized: now 1790637450188222 us; error 31 us; skew 500 ppm
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:30.190439 7854 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 23:17:30.190837 7854 webserver.cc:460] Webserver started at http://127.0.98.62:46225/ using document root <none> and password file <none>
I20260928 23:17:30.191020 7854 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:30.191071 7854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:30.191200 7854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:30.192211 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "163b0b9acfa5432fb81babb620610e08"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "1924a797be821c9c185a942e4b4f616b"
server_key_iv: "eb98d1a99835b1647f55c6c926fb730d"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.192528 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "163b0b9acfa5432fb81babb620610e08"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "1924a797be821c9c185a942e4b4f616b"
server_key_iv: "eb98d1a99835b1647f55c6c926fb730d"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.193795 7854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:30.194425 7869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:30.194574 7854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:30.194651 7854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "163b0b9acfa5432fb81babb620610e08"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "1924a797be821c9c185a942e4b4f616b"
server_key_iv: "eb98d1a99835b1647f55c6c926fb730d"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.194710 7854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:30.211246 7854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:30.213024 7854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:30.213166 7854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:30.217245 7854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45297
I20260928 23:17:30.217307 7931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45297 every 8 connection(s)
I20260928 23:17:30.217581 7854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:30.218186 7932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:30.220409 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08: Bootstrap starting.
I20260928 23:17:30.220937 7932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:30.221186 7932 log.cc:824] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:30.221777 7932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08: No bootstrap required, opened a new log
I20260928 23:17:30.223760 7932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b0b9acfa5432fb81babb620610e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260928 23:17:30.223899 7932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:30.223920 7932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 163b0b9acfa5432fb81babb620610e08, State: Initialized, Role: FOLLOWER
I20260928 23:17:30.224044 7932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b0b9acfa5432fb81babb620610e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260928 23:17:30.224103 7932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:30.224124 7932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:30.224170 7932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:30.224653 7932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b0b9acfa5432fb81babb620610e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260928 23:17:30.224738 7932 leader_election.cc:302] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 163b0b9acfa5432fb81babb620610e08; no voters:
I20260928 23:17:30.224886 7932 leader_election.cc:288] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:30.224931 7935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:30.225100 7932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:30.225104 7935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [term 1 LEADER]: Becoming Leader. State: Replica: 163b0b9acfa5432fb81babb620610e08, State: Running, Role: LEADER
I20260928 23:17:30.225229 7935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b0b9acfa5432fb81babb620610e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } }
I20260928 23:17:30.225577 7936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "163b0b9acfa5432fb81babb620610e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b0b9acfa5432fb81babb620610e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } } }
I20260928 23:17:30.225589 7937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 163b0b9acfa5432fb81babb620610e08. Latest consensus state: current_term: 1 leader_uuid: "163b0b9acfa5432fb81babb620610e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "163b0b9acfa5432fb81babb620610e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45297 } } }
I20260928 23:17:30.225677 7936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:30.225687 7937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:30.226099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 7854
I20260928 23:17:30.226208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:30.226284 7944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:30.226418 392 external_mini_cluster.cc:1428] Setting key 330e8dbd94a836b63270be0461654b41
I20260928 23:17:30.227057 7944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:30.231982 7944 catalog_manager.cc:1347] Generated new cluster ID: 47f05759727541e6b456e349ab4b190a
I20260928 23:17:30.232039 7944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:30.237493 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.226822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.0.98.62:45297)
0928 23:17:30.227645 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.227655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.227687 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:17:30.227796 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.227812 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.227865 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.228004 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:30.228583 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.230010 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.230718 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.231337 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.231440 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.234102 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:30.234106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:30.234107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:30.234118 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:30.236002 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.236372 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.236375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.236375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.236471 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.236650 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.236650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.236651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.236922 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:30.237058 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.237153 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.237278 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":67,"threads_started":1}
I20260928 23:17:30.248188 7944 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:30.248680 7944 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:30.254642 7944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08: Generated new TSK 0
I20260928 23:17:30.254812 7944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 23:17:30.286394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:30.364154 7958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:30.364347 7958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:30.364368 7958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:30.365468 7958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:30.365504 7958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:30.365521 7958 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:17:30.365537 7958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:30.366940 7958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:30.367111 7958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:30.367278 7958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:30.371851 7966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:30.371948 7958 server_base.cc:1034] running on GCE node
W20260928 23:17:30.371886 7963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:30.371929 7964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:30.372219 7958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:30.372433 7958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:30.373574 7958 hybrid_clock.cc:648] HybridClock initialized: now 1790637450373553 us; error 35 us; skew 500 ppm
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:30.375823 7958 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 23:17:30.376243 7958 webserver.cc:460] Webserver started at http://127.0.98.1:37023/ using document root <none> and password file <none>
I20260928 23:17:30.376441 7958 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:30.376487 7958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:30.376607 7958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:30.377569 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "d79e7bad34484c31864d4948fc8d8a51"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "42a8a2447b7a6bb39ccbd27ca1fbc30c"
server_key_iv: "0af0001d44e24834caf853a9c4cb30d7"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.377920 7958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d79e7bad34484c31864d4948fc8d8a51"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "42a8a2447b7a6bb39ccbd27ca1fbc30c"
server_key_iv: "0af0001d44e24834caf853a9c4cb30d7"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.379113 7958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:17:30.379838 7973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:30.379989 7958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:30.380074 7958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "d79e7bad34484c31864d4948fc8d8a51"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "42a8a2447b7a6bb39ccbd27ca1fbc30c"
server_key_iv: "0af0001d44e24834caf853a9c4cb30d7"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.380142 7958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:30.401957 7958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:30.404047 7958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:30.404188 7958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:30.404716 7958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:30.404764 7958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:30.404812 7958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:30.404836 7958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:30.411391 7958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35365
I20260928 23:17:30.411469 8088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35365 every 8 connection(s)
I20260928 23:17:30.411758 7958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:30.413537 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.405137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56869 (local address 127.0.98.62:45297)
0928 23:17:30.405188 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.405190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.405373 (+ 183us) server_negotiation.cc:407] Connection header received
0928 23:17:30.405873 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.405874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.405889 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.405941 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:30.406560 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.407324 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.408368 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.408574 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.408628 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.410912 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:30.410914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:30.410915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:30.410924 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:30.412425 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.412789 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.412791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.412791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.412848 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.413031 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.413032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.413032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.413126 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:30.413197 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.413409 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.413474 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:17:30.413659 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.405105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:56869)
0928 23:17:30.405260 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.405295 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.405782 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.406012 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.406016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.406288 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:30.406472 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.406477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.407416 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.407418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.408285 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.408287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.408642 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.408643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.408782 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.409210 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:30.409220 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:30.410819 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:30.412487 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.412490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.412492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.412691 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.412901 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.412902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.412902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.412989 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.413189 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:30.413192 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:30.413295 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.413390 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.413500 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260928 23:17:30.414116 8089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45297
I20260928 23:17:30.414191 8089 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:30.414372 8089 heartbeater.cc:511] Master 127.0.98.62:45297 requested a full tablet report, sending...
I20260928 23:17:30.414875 7885 ts_manager.cc:194] Registered new tserver with Master: d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365)
I20260928 23:17:30.415700 7885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56869
I20260928 23:17:30.419677 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.415094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43227 (local address 127.0.98.62:45297)
0928 23:17:30.415181 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.415183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.415215 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:17:30.415257 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.415258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.415267 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.415318 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:30.415572 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.416187 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.416861 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.417057 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.417108 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.417811 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:30.417812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:30.417812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:30.417819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:30.418769 (+ 950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.419035 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.419036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.419036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.419085 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.419238 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.419239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.419240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.419336 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:30.419387 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.419511 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.419598 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:17:30.419700 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.415046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.1:43227)
0928 23:17:30.415167 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.415175 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.415223 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.415320 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.415321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.415463 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:30.415508 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.415509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.416275 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.416276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.416794 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.416795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.417112 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.417113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.417173 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.417488 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:30.417492 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:30.417741 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:30.418835 (+ 1094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.418836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.418836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.418981 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.419135 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.419135 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.419136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.419189 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.419386 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:30.419386 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:30.419427 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.419486 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.419593 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260928 23:17:30.420697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 7958
I20260928 23:17:30.420815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:30.421020 392 external_mini_cluster.cc:1428] Setting key 6882886e51504199b6e1f8568bd1e926
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 23:17:30.442237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:30.522264 8098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:30.522472 8098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:30.522496 8098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:30.523626 8098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:30.523660 8098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:30.523679 8098 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:17:30.523702 8098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:30.525216 8098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:30.525410 8098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:30.525583 8098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:17:30.530467 8098 server_base.cc:1034] running on GCE node
W20260928 23:17:30.530447 8103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:30.530494 8104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:30.530448 8106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:30.530858 8098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:30.531087 8098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:30.532217 8098 hybrid_clock.cc:648] HybridClock initialized: now 1790637450532203 us; error 32 us; skew 500 ppm
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:30.534502 8098 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 23:17:30.534911 8098 webserver.cc:460] Webserver started at http://127.0.98.2:40695/ using document root <none> and password file <none>
I20260928 23:17:30.535094 8098 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:30.535132 8098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:30.535221 8098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:30.536245 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "f838121d0c3a42c6a46f4074b0e0bcf3"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "388d8867961f25eec05e3be312daa43c"
server_key_iv: "3120f6edc21338276f6e04ae2a4c25a2"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.536538 8098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f838121d0c3a42c6a46f4074b0e0bcf3"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "388d8867961f25eec05e3be312daa43c"
server_key_iv: "3120f6edc21338276f6e04ae2a4c25a2"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.537683 8098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:30.538334 8113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:30.538476 8098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:30.538555 8098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "f838121d0c3a42c6a46f4074b0e0bcf3"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "388d8867961f25eec05e3be312daa43c"
server_key_iv: "3120f6edc21338276f6e04ae2a4c25a2"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.538614 8098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:30.563926 8098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:30.565987 8098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:30.566120 8098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:30.566672 8098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:30.566720 8098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:30.566746 8098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:30.566768 8098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:30.573566 8098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44401
I20260928 23:17:30.573891 8098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:30.573639 8228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44401 every 8 connection(s)
I20260928 23:17:30.575766 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.567065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41441 (local address 127.0.98.62:45297)
0928 23:17:30.567126 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.567129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.567333 (+ 204us) server_negotiation.cc:407] Connection header received
0928 23:17:30.568077 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.568079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.568093 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.568161 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:30.568969 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.569477 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.570933 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.571132 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.571181 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.573461 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:30.573462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:30.573462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:30.573468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:30.574746 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.575069 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.575070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.575070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.575112 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.575269 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.575269 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.575270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.575359 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:30.575405 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.575644 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.575703 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:17:30.575942 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.567041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:41441)
0928 23:17:30.567192 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.567240 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.567940 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.568252 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.568256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.568601 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:30.568886 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.568894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.569546 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.569547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.570847 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.570850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.571211 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.571211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.571371 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.571881 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:30.571890 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:30.573361 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:30.574818 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.574821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.574822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.575012 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.575146 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.575147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.575147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.575229 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.575403 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:30.575406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:30.575509 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.575624 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.575737 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260928 23:17:30.576283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 8098
I20260928 23:17:30.576395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:30.576421 8229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45297
I20260928 23:17:30.576532 8229 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:30.576591 392 external_mini_cluster.cc:1428] Setting key 12a7a24dbc350fc4ea7411c938f08e16
I20260928 23:17:30.576704 8229 heartbeater.cc:511] Master 127.0.98.62:45297 requested a full tablet report, sending...
I20260928 23:17:30.577013 7885 ts_manager.cc:194] Registered new tserver with Master: f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401)
I20260928 23:17:30.577415 7885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41441
I20260928 23:17:30.584000 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.578671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.2:50461)
0928 23:17:30.578801 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.578811 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.578859 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.578967 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.578968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.579099 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:30.579151 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.579152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.579961 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.579961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.580542 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.580545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.580959 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.580960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.580999 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.581350 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:30.581354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:30.581596 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:30.583062 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.583063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.583063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.583245 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.583412 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.583412 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.583412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.583458 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.583780 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:30.583781 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:30.583816 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.583867 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.583933 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260928 23:17:30.584038 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.578745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50461 (local address 127.0.98.62:45297)
0928 23:17:30.578786 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.578788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.578850 (+ 62us) server_negotiation.cc:407] Connection header received
0928 23:17:30.578907 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.578907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.578916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.578961 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:30.579205 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.579886 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.580622 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.580896 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.580963 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.581672 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:30.581674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:30.581674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:30.581680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:30.582993 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.583307 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.583308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.583309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.583358 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.583506 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.583506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.583507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.583718 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:30.583789 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.583891 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.583955 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 23:17:30.601897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45297
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:30.678536 8238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:30.678726 8238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:30.678746 8238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:30.679849 8238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:30.679885 8238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:30.679904 8238 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:17:30.679934 8238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:30.681367 8238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:30.681565 8238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:30.681763 8238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:30.686583 8244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:30.686618 8243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:30.686646 8238 server_base.cc:1034] running on GCE node
W20260928 23:17:30.686609 8246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:30.686951 8238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:30.687163 8238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:30.688294 8238 hybrid_clock.cc:648] HybridClock initialized: now 1790637450688256 us; error 55 us; skew 500 ppm
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:30.690543 8238 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 23:17:30.690979 8238 webserver.cc:460] Webserver started at http://127.0.98.3:43473/ using document root <none> and password file <none>
I20260928 23:17:30.691145 8238 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:30.691184 8238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:30.691267 8238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:30.692286 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "9ffc81c6d96c4e97b655efdb67bcee5e"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "e9a6ddc873414aa9fa9c115a4d6a7567"
server_key_iv: "181a2f919e7e9da499547b50a03c376a"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.692575 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9ffc81c6d96c4e97b655efdb67bcee5e"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "e9a6ddc873414aa9fa9c115a4d6a7567"
server_key_iv: "181a2f919e7e9da499547b50a03c376a"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.693743 8238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:30.694443 8253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:30.694595 8238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:30.694664 8238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "9ffc81c6d96c4e97b655efdb67bcee5e"
format_stamp: "Formatted at 2026-09-28 23:17:30 on dist-test-slave-djkr"
server_key: "e9a6ddc873414aa9fa9c115a4d6a7567"
server_key_iv: "181a2f919e7e9da499547b50a03c376a"
server_key_version: "encryptionkey@0"
I20260928 23:17:30.694730 8238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:30.715055 8238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:30.717032 8238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:30.717146 8238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:30.717634 8238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:30.717674 8238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:30.717694 8238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:30.717705 8238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:30.724231 8238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33853
I20260928 23:17:30.724303 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33853 every 8 connection(s)
I20260928 23:17:30.724586 8238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:30.726054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 8238
I20260928 23:17:30.726153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:30.726375 392 external_mini_cluster.cc:1428] Setting key c38cf7e2596b6083d0b63b7067405f4d
I20260928 23:17:30.727087 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.718046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43685 (local address 127.0.98.62:45297)
0928 23:17:30.718107 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.718109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.718346 (+ 237us) server_negotiation.cc:407] Connection header received
0928 23:17:30.718995 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.718997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.719007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.719045 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:30.719889 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.720407 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.721401 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.721589 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.721650 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.724191 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:30.724193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:30.724193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:30.724201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:30.725641 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.726012 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.726013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.726014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.726078 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.726377 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.726377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.726378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.726512 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:30.726574 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.726924 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.727002 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:17:30.727239 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.718025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:43685)
0928 23:17:30.718183 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.718232 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.718894 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.719129 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.719133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.719482 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:30.719793 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.719801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.720494 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.720495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.721314 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.721317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.721749 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.721751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.721862 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.722283 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:30.722292 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:30.724102 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:30.725710 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.725714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.725716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.725940 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.726200 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.726201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.726201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.726316 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.726607 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:30.726613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:30.726765 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.726890 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.727038 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260928 23:17:30.727648 8369 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45297
I20260928 23:17:30.727720 8369 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:30.727881 8369 heartbeater.cc:511] Master 127.0.98.62:45297 requested a full tablet report, sending...
I20260928 23:17:30.728196 7885 ts_manager.cc:194] Registered new tserver with Master: 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3:33853)
I20260928 23:17:30.728654 7885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43685
I20260928 23:17:30.733513 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.728574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41559 (local address 127.0.98.62:45297)
0928 23:17:30.728611 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.728613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.728672 (+ 59us) server_negotiation.cc:407] Connection header received
0928 23:17:30.728741 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.728742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.728752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.728812 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:30.729082 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.729642 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.730385 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.730685 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.730748 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.731596 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:30.731597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:30.731597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:30.731603 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:30.732596 (+ 993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.732883 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.732884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.732885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.732935 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.733085 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.733085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.733086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.733184 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:30.733234 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.733359 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.733425 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:17:30.733527 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.728496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.98.3:41559)
0928 23:17:30.728609 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.728627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.728686 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.728811 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.728812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.728965 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:30.729018 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.729019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.729712 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.729713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.730312 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.730314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.730750 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.730751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.730840 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.731253 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:30.731258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:30.731504 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:30.732679 (+ 1175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.732680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.732681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.732828 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.732984 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:30.732985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:30.732985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:30.733038 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:30.733234 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:30.733234 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:30.733275 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.733333 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.733426 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260928 23:17:30.737938 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:17:30.757057 7954 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:30.752249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.0.98.62:45297)
0928 23:17:30.752301 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.752303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.752315 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:17:30.752372 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.752373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.752383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.752427 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:30.752937 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.753403 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.754096 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.754275 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.754326 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.756970 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60368: BlockingRecv error: recv got EOF from 127.0.0.1:60368 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 23:17:30 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:30.763567 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.757691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.0.98.62:45297)
0928 23:17:30.757751 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.757752 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.757761 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:30.757796 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.757796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.757803 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.757844 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:30.758267 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.758709 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.759372 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.759542 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.759643 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.761424 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:30.761426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:30.761426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:30.761432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:30.762685 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.762986 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.762987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.762987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.763035 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:30.763211 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:30.763212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:30.763212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:30.763297 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:30.763361 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.763437 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.763485 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:17:30.764971 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:30.766484 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:30.772627 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.768573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44401 (local address 127.0.0.1:54856)
0928 23:17:30.768942 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.768948 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.768977 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.769182 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.769185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.769187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:30.769261 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.769263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.770583 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.770584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.771693 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.771695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.772324 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.772325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.772407 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.772422 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.772486 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.772536 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":31,"thread_start_us":86,"threads_started":1}
I20260928 23:17:30.772711 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.768678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.0.98.1:35365)
0928 23:17:30.769235 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.769239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.769248 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:30.769282 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.769284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.769299 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.769369 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:30.769610 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.770273 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.771734 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.772422 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.772480 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.772555 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.772586 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.772627 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":74,"threads_started":1}
I20260928 23:17:30.772888 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.768803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.0.98.2:44401)
0928 23:17:30.768988 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.768993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.769005 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:17:30.769043 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.769045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.769060 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.769158 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:30.769353 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.770503 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.771771 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.772248 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.772309 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.772391 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.772742 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.772790 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":121,"threads_started":1}
I20260928 23:17:30.772904 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.769022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.0.98.3:33853)
0928 23:17:30.769200 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.769205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.769213 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:30.769268 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.769270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.769285 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.769475 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:30.769532 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.770540 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.771599 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.772301 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.772681 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.772761 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.772786 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.772823 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260928 23:17:30.772639 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.768573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33853 (local address 127.0.0.1:54926)
0928 23:17:30.768833 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.768841 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.768879 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.769374 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.769375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.769377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:30.769416 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.769417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.770635 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.770636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.771498 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.771500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.772398 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.772399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.772469 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.772473 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.772501 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.772540 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
I20260928 23:17:30.773353 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.768565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35365 (local address 127.0.0.1:50552)
0928 23:17:30.768727 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.768768 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.768831 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.769465 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.769466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.769468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:30.769522 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.769523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.770351 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.770351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.771648 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.771651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.772482 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.772483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.772534 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.772537 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.772574 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.772609 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":33,"thread_start_us":65,"threads_started":1}
I20260928 23:17:30.773671 8152 tablet_service.cc:1467] Processing CreateTablet for tablet c32d314d6f8645be894b06e4550c75fc (DEFAULT_TABLE table=test-table [id=aca509e3edce4763b22bd171260ce61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:30.773973 8152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32d314d6f8645be894b06e4550c75fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:30.773903 8023 tablet_service.cc:1467] Processing CreateTablet for tablet c32d314d6f8645be894b06e4550c75fc (DEFAULT_TABLE table=test-table [id=aca509e3edce4763b22bd171260ce61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:30.773901 8303 tablet_service.cc:1467] Processing CreateTablet for tablet c32d314d6f8645be894b06e4550c75fc (DEFAULT_TABLE table=test-table [id=aca509e3edce4763b22bd171260ce61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:30.774220 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32d314d6f8645be894b06e4550c75fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:30.774219 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32d314d6f8645be894b06e4550c75fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:30.776958 8397 tablet_bootstrap.cc:492] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: Bootstrap starting.
I20260928 23:17:30.777601 8398 tablet_bootstrap.cc:492] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: Bootstrap starting.
I20260928 23:17:30.777784 8397 tablet_bootstrap.cc:654] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:30.777823 8399 tablet_bootstrap.cc:492] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: Bootstrap starting.
I20260928 23:17:30.778191 8398 tablet_bootstrap.cc:654] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:30.778198 8397 log.cc:824] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:30.778349 8399 tablet_bootstrap.cc:654] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:30.778501 8398 log.cc:824] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:30.778666 8399 log.cc:824] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:30.778976 8397 tablet_bootstrap.cc:492] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: No bootstrap required, opened a new log
I20260928 23:17:30.779047 8397 ts_tablet_manager.cc:1397] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:30.779130 8398 tablet_bootstrap.cc:492] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: No bootstrap required, opened a new log
I20260928 23:17:30.779198 8398 ts_tablet_manager.cc:1397] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:30.779462 8399 tablet_bootstrap.cc:492] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: No bootstrap required, opened a new log
I20260928 23:17:30.779524 8399 ts_tablet_manager.cc:1397] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:30.780634 8397 raft_consensus.cc:348] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } }
I20260928 23:17:30.780762 8397 raft_consensus.cc:374] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:30.780792 8397 raft_consensus.cc:729] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f838121d0c3a42c6a46f4074b0e0bcf3, State: Initialized, Role: FOLLOWER
I20260928 23:17:30.780928 8397 consensus_queue.cc:260] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } }
I20260928 23:17:30.781060 8398 raft_consensus.cc:348] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } }
I20260928 23:17:30.781184 8229 heartbeater.cc:503] Master 127.0.98.62:45297 was elected leader, sending a full tablet report...
I20260928 23:17:30.781224 8398 raft_consensus.cc:374] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:30.781265 8398 raft_consensus.cc:729] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffc81c6d96c4e97b655efdb67bcee5e, State: Initialized, Role: FOLLOWER
I20260928 23:17:30.781184 8397 ts_tablet_manager.cc:1428] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:17:30.781410 8398 consensus_queue.cc:260] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } }
I20260928 23:17:30.781388 8399 raft_consensus.cc:348] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } }
I20260928 23:17:30.781528 8399 raft_consensus.cc:374] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:30.781572 8399 raft_consensus.cc:729] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79e7bad34484c31864d4948fc8d8a51, State: Initialized, Role: FOLLOWER
I20260928 23:17:30.781661 8398 ts_tablet_manager.cc:1428] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:17:30.781677 8399 consensus_queue.cc:260] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } }
I20260928 23:17:30.781774 8369 heartbeater.cc:503] Master 127.0.98.62:45297 was elected leader, sending a full tablet report...
I20260928 23:17:30.782199 8399 ts_tablet_manager.cc:1428] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:17:30.782226 8089 heartbeater.cc:503] Master 127.0.98.62:45297 was elected leader, sending a full tablet report...
W20260928 23:17:30.824618 8230 tablet.cc:2369] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:30.909080 8404 raft_consensus.cc:482] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:30.909205 8404 raft_consensus.cc:504] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } }
I20260928 23:17:30.909550 8404 leader_election.cc:288] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401), d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365)
I20260928 23:17:30.912148 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.909826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49803 (local address 127.0.98.2:44401)
0928 23:17:30.909869 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.909870 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.909877 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:30.909904 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.909905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.909912 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.909955 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:30.910096 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.910557 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.911457 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.911856 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.911915 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.911926 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.912051 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.912087 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:17:30.912149 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.909731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44401 (local address 127.0.98.3:49803)
0928 23:17:30.909772 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.909784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.909828 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.909972 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.909972 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.909974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:30.910038 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.910040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.910630 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.910631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.911373 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.911375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.911924 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.911925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.911995 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.911998 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.912042 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.912080 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 23:17:30.912212 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.909807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60823 (local address 127.0.98.1:35365)
0928 23:17:30.909850 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.909852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.909987 (+ 135us) server_negotiation.cc:407] Connection header received
0928 23:17:30.910033 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.910034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.910046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.910092 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:30.910242 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.910697 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.911577 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.911916 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.911973 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.911982 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.912108 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.912150 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:17:30.912263 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.909733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35365 (local address 127.0.98.3:60823)
0928 23:17:30.909917 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:30.909926 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:30.909983 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:30.910101 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:30.910101 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:30.910103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:30.910181 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.910182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.910769 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.910769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.911475 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:30.911477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.911982 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.911983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:30.912051 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.912054 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:30.912098 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:17:30.912138 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":28,"thread_start_us":108,"threads_started":1}
I20260928 23:17:30.912500 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32d314d6f8645be894b06e4550c75fc" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" is_pre_election: true
W20260928 23:17:30.912549 8090 tablet.cc:2369] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:30.912556 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32d314d6f8645be894b06e4550c75fc" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e7bad34484c31864d4948fc8d8a51" is_pre_election: true
I20260928 23:17:30.912654 8183 raft_consensus.cc:2454] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 0.
I20260928 23:17:30.912699 8043 raft_consensus.cc:2454] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 0.
I20260928 23:17:30.912896 8254 leader_election.cc:302] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ffc81c6d96c4e97b655efdb67bcee5e, d79e7bad34484c31864d4948fc8d8a51; no voters:
I20260928 23:17:30.913025 8404 raft_consensus.cc:2781] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:30.913081 8404 raft_consensus.cc:482] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:30.913097 8404 raft_consensus.cc:3037] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:30.914045 8404 raft_consensus.cc:504] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } }
I20260928 23:17:30.914161 8404 leader_election.cc:288] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 election: Requested vote from peers f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401), d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365)
I20260928 23:17:30.914305 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32d314d6f8645be894b06e4550c75fc" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3"
I20260928 23:17:30.914384 8183 raft_consensus.cc:3037] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:30.914430 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32d314d6f8645be894b06e4550c75fc" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e7bad34484c31864d4948fc8d8a51"
I20260928 23:17:30.914521 8043 raft_consensus.cc:3037] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:30.915117 8183 raft_consensus.cc:2454] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 1.
I20260928 23:17:30.915246 8043 raft_consensus.cc:2454] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 1.
I20260928 23:17:30.915295 8255 leader_election.cc:302] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ffc81c6d96c4e97b655efdb67bcee5e, f838121d0c3a42c6a46f4074b0e0bcf3; no voters:
I20260928 23:17:30.915371 8404 raft_consensus.cc:2781] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:30.915521 8404 raft_consensus.cc:686] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 LEADER]: Becoming Leader. State: Replica: 9ffc81c6d96c4e97b655efdb67bcee5e, State: Running, Role: LEADER
I20260928 23:17:30.915663 8404 consensus_queue.cc:237] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } }
I20260928 23:17:30.916399 7885 catalog_manager.cc:5495] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } health_report { overall_health: UNKNOWN } } }
I20260928 23:17:30.955513 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:30.953425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.0.98.3:33853)
0928 23:17:30.953476 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:30.953478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:30.953485 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:30.953518 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:30.953518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:30.953537 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:30.953596 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:30.953711 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.954165 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.954877 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:30.955053 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:30.955095 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:30.955262 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:30.955358 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:30.955409 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:30.955437 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 23:17:30.955453 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:17:30.957383 8183 raft_consensus.cc:1261] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ffc81c6d96c4e97b655efdb67bcee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:30.957392 8043 raft_consensus.cc:1261] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Refusing update from remote peer 9ffc81c6d96c4e97b655efdb67bcee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:30.957702 8404 consensus_queue.cc:1035] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:30.957918 8407 consensus_queue.cc:1035] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:30.960568 8411 mvcc.cc:205] Tried to move back new op lower bound from 7334450999118049280 to 7334450998951092224. Current Snapshot: MvccSnapshot[applied={T|T < 7334450999118049280}]
I20260928 23:17:30.961952 7886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:30.962051 7886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:30.962711 7886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08: Sending DeleteTablet for 3 replicas of tablet c32d314d6f8645be894b06e4550c75fc
I20260928 23:17:30.963004 8152 tablet_service.cc:1514] Processing DeleteTablet for tablet c32d314d6f8645be894b06e4550c75fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260928 23:17:30.963331 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet c32d314d6f8645be894b06e4550c75fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50552
I20260928 23:17:30.963418 7886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:30.963572 7886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:30.963586 8421 tablet_replica.cc:330] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: stopping tablet replica
I20260928 23:17:30.963584 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet c32d314d6f8645be894b06e4550c75fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54926
I20260928 23:17:30.963678 8421 raft_consensus.cc:2229] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:30.963768 8421 raft_consensus.cc:2258] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:30.964641 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 5543676df57a40b293fdc94442665c6d (DEFAULT_TABLE table=test-table [id=eb2332461f774a8f8c6959c79b69e4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:30.964689 8152 tablet_service.cc:1467] Processing CreateTablet for tablet 5543676df57a40b293fdc94442665c6d (DEFAULT_TABLE table=test-table [id=eb2332461f774a8f8c6959c79b69e4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:30.964743 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5543676df57a40b293fdc94442665c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:30.964774 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 5543676df57a40b293fdc94442665c6d (DEFAULT_TABLE table=test-table [id=eb2332461f774a8f8c6959c79b69e4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:30.964797 8152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5543676df57a40b293fdc94442665c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:30.964860 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5543676df57a40b293fdc94442665c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:30.966032 8399 tablet_bootstrap.cc:492] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51: Bootstrap starting.
I20260928 23:17:30.966091 8398 tablet_bootstrap.cc:492] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: Bootstrap starting.
I20260928 23:17:30.964568 8421 ts_tablet_manager.cc:1905] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:30.966391 8399 tablet_bootstrap.cc:654] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:30.966460 8398 tablet_bootstrap.cc:654] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:30.966773 8397 tablet_bootstrap.cc:492] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3: Bootstrap starting.
I20260928 23:17:30.967011 8399 tablet_bootstrap.cc:492] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51: No bootstrap required, opened a new log
I20260928 23:17:30.967063 8399 ts_tablet_manager.cc:1397] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:30.967129 8397 tablet_bootstrap.cc:654] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:30.967203 8399 raft_consensus.cc:348] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:30.967276 8399 raft_consensus.cc:374] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:30.967296 8399 raft_consensus.cc:729] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79e7bad34484c31864d4948fc8d8a51, State: Initialized, Role: FOLLOWER
I20260928 23:17:30.967348 8399 consensus_queue.cc:260] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:30.967634 8399 ts_tablet_manager.cc:1428] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:30.967796 8397 tablet_bootstrap.cc:492] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3: No bootstrap required, opened a new log
I20260928 23:17:30.967835 8397 ts_tablet_manager.cc:1397] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:30.967975 8397 raft_consensus.cc:348] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:30.968035 8397 raft_consensus.cc:374] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:30.968055 8397 raft_consensus.cc:729] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f838121d0c3a42c6a46f4074b0e0bcf3, State: Initialized, Role: FOLLOWER
I20260928 23:17:30.968091 8397 consensus_queue.cc:260] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:30.968170 8397 ts_tablet_manager.cc:1428] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:30.968348 8421 ts_tablet_manager.cc:1918] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:30.968516 8421 log.cc:1196] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/c32d314d6f8645be894b06e4550c75fc
I20260928 23:17:30.968762 8421 ts_tablet_manager.cc:1939] T c32d314d6f8645be894b06e4550c75fc P d79e7bad34484c31864d4948fc8d8a51: Deleting consensus metadata
I20260928 23:17:30.969141 7870 catalog_manager.cc:4841] TS d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365): tablet c32d314d6f8645be894b06e4550c75fc (table test-table [id=aca509e3edce4763b22bd171260ce61b]) successfully deleted
I20260928 23:17:30.971858 8420 tablet_replica.cc:330] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: stopping tablet replica
I20260928 23:17:30.971940 8398 tablet_bootstrap.cc:492] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: No bootstrap required, opened a new log
I20260928 23:17:30.971962 8420 raft_consensus.cc:2229] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:30.971997 8398 ts_tablet_manager.cc:1397] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 23:17:30.972050 8420 raft_consensus.cc:2258] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:30.972149 8398 raft_consensus.cc:348] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:30.972205 8422 tablet_replica.cc:330] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: stopping tablet replica
I20260928 23:17:30.972229 8398 raft_consensus.cc:374] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:30.972255 8398 raft_consensus.cc:729] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffc81c6d96c4e97b655efdb67bcee5e, State: Initialized, Role: FOLLOWER
I20260928 23:17:30.972294 8398 consensus_queue.cc:260] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:30.972383 8398 ts_tablet_manager.cc:1428] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:30.972620 8420 ts_tablet_manager.cc:1905] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:30.972298 8422 raft_consensus.cc:2229] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:30.974143 8420 ts_tablet_manager.cc:1918] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:30.974161 8422 raft_consensus.cc:2258] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:30.974242 8420 log.cc:1196] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/c32d314d6f8645be894b06e4550c75fc
I20260928 23:17:30.974481 8420 ts_tablet_manager.cc:1939] T c32d314d6f8645be894b06e4550c75fc P f838121d0c3a42c6a46f4074b0e0bcf3: Deleting consensus metadata
I20260928 23:17:30.974563 8422 ts_tablet_manager.cc:1905] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:30.974830 7871 catalog_manager.cc:4841] TS f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401): tablet c32d314d6f8645be894b06e4550c75fc (table test-table [id=aca509e3edce4763b22bd171260ce61b]) successfully deleted
W20260928 23:17:30.975308 8370 tablet.cc:2369] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:30.975957 8422 ts_tablet_manager.cc:1918] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:30.976028 8422 log.cc:1196] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/c32d314d6f8645be894b06e4550c75fc
I20260928 23:17:30.976265 8422 ts_tablet_manager.cc:1939] T c32d314d6f8645be894b06e4550c75fc P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting consensus metadata
I20260928 23:17:30.976619 7872 catalog_manager.cc:4841] TS 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3:33853): tablet c32d314d6f8645be894b06e4550c75fc (table test-table [id=aca509e3edce4763b22bd171260ce61b]) successfully deleted
I20260928 23:17:31.169997 8417 raft_consensus.cc:482] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:31.170085 8417 raft_consensus.cc:504] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.170246 8417 leader_election.cc:288] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365), f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401)
I20260928 23:17:31.170486 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5543676df57a40b293fdc94442665c6d" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e7bad34484c31864d4948fc8d8a51" is_pre_election: true
I20260928 23:17:31.170477 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5543676df57a40b293fdc94442665c6d" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" is_pre_election: true
I20260928 23:17:31.170588 8183 raft_consensus.cc:2454] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 0.
I20260928 23:17:31.170588 8043 raft_consensus.cc:2454] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 0.
I20260928 23:17:31.170747 8255 leader_election.cc:302] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ffc81c6d96c4e97b655efdb67bcee5e, f838121d0c3a42c6a46f4074b0e0bcf3; no voters:
I20260928 23:17:31.170872 8417 raft_consensus.cc:2781] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:31.170926 8417 raft_consensus.cc:482] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:31.170941 8417 raft_consensus.cc:3037] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.171645 8417 raft_consensus.cc:504] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.171761 8417 leader_election.cc:288] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 election: Requested vote from peers d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365), f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401)
I20260928 23:17:31.171911 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5543676df57a40b293fdc94442665c6d" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3"
I20260928 23:17:31.171929 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5543676df57a40b293fdc94442665c6d" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e7bad34484c31864d4948fc8d8a51"
I20260928 23:17:31.171985 8183 raft_consensus.cc:3037] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.171996 8043 raft_consensus.cc:3037] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.172560 8043 raft_consensus.cc:2454] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 1.
I20260928 23:17:31.172693 8183 raft_consensus.cc:2454] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 1.
I20260928 23:17:31.172715 8254 leader_election.cc:302] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ffc81c6d96c4e97b655efdb67bcee5e, d79e7bad34484c31864d4948fc8d8a51; no voters:
I20260928 23:17:31.172787 8417 raft_consensus.cc:2781] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:31.172851 8417 raft_consensus.cc:686] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 LEADER]: Becoming Leader. State: Replica: 9ffc81c6d96c4e97b655efdb67bcee5e, State: Running, Role: LEADER
I20260928 23:17:31.172909 8417 consensus_queue.cc:237] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.173462 7886 catalog_manager.cc:5495] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } health_report { overall_health: HEALTHY } } }
Sep 28 23:17:31 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637450, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:31.226086 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.220139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45297 (local address 127.0.0.1:60382)
0928 23:17:31.220263 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:31.220272 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:31.220331 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:31.220436 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:31.220436 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:31.220707 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:31.220750 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:31.220751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.221325 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.221326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:31.221835 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:31.221837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.222123 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.222124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:31.222156 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.222617 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:31.222621 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:31.223935 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:31.225166 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:31.225173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:31.225174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:31.225371 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:31.225537 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:31.225538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:31.225538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:31.225619 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:31.225820 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:31.225824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:31.225861 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:17:31.225908 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:17:31.225988 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260928 23:17:31.226125 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.220171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.0.98.62:45297)
0928 23:17:31.220222 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:31.220224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:31.220328 (+ 104us) server_negotiation.cc:407] Connection header received
0928 23:17:31.220371 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:31.220372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:31.220380 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:31.220425 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:31.220801 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.221270 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.221897 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.222074 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.222124 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.224001 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:31.224002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:31.224002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:31.224008 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:31.225090 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:31.225429 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:31.225429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:31.225430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:31.225486 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:31.225672 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:31.225673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:31.225673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:31.225777 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:31.225820 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:31.225931 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 23:17:31.226032 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":28}
I20260928 23:17:31.227902 7886 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60382:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:17:31.228081 7886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:31.229331 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 3ace2b11451648088bac04c82d73d3d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99bac4ee64d642d3a20b7e4f986f8786]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:17:31.229365 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 3ace2b11451648088bac04c82d73d3d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99bac4ee64d642d3a20b7e4f986f8786]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:17:31.229386 8152 tablet_service.cc:1467] Processing CreateTablet for tablet 3ace2b11451648088bac04c82d73d3d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=99bac4ee64d642d3a20b7e4f986f8786]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:17:31.229476 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ace2b11451648088bac04c82d73d3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:31.229480 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ace2b11451648088bac04c82d73d3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:31.229493 8152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ace2b11451648088bac04c82d73d3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:31.230482 8399 tablet_bootstrap.cc:492] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: Bootstrap starting.
I20260928 23:17:31.230842 8399 tablet_bootstrap.cc:654] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:31.230852 8397 tablet_bootstrap.cc:492] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: Bootstrap starting.
I20260928 23:17:31.231179 8397 tablet_bootstrap.cc:654] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:31.231424 8399 tablet_bootstrap.cc:492] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: No bootstrap required, opened a new log
I20260928 23:17:31.231472 8399 ts_tablet_manager.cc:1397] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:31.231473 8398 tablet_bootstrap.cc:492] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: Bootstrap starting.
I20260928 23:17:31.231635 8399 raft_consensus.cc:348] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260928 23:17:31.231710 8399 raft_consensus.cc:374] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:31.231730 8399 raft_consensus.cc:729] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79e7bad34484c31864d4948fc8d8a51, State: Initialized, Role: FOLLOWER
I20260928 23:17:31.231784 8399 consensus_queue.cc:260] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260928 23:17:31.231845 8398 tablet_bootstrap.cc:654] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:31.231891 8399 ts_tablet_manager.cc:1428] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:31.232014 8405 tablet_replica.cc:439] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } }
I20260928 23:17:31.232095 8405 tablet_replica.cc:442] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:31.232194 8397 tablet_bootstrap.cc:492] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: No bootstrap required, opened a new log
I20260928 23:17:31.232249 8397 ts_tablet_manager.cc:1397] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:31.232390 8397 raft_consensus.cc:348] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260928 23:17:31.232457 8397 raft_consensus.cc:374] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:31.232481 8397 raft_consensus.cc:729] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f838121d0c3a42c6a46f4074b0e0bcf3, State: Initialized, Role: FOLLOWER
I20260928 23:17:31.232517 8397 consensus_queue.cc:260] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260928 23:17:31.232599 8397 ts_tablet_manager.cc:1428] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:31.232666 8403 tablet_replica.cc:439] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } }
I20260928 23:17:31.232756 8403 tablet_replica.cc:442] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:31.232813 8398 tablet_bootstrap.cc:492] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: No bootstrap required, opened a new log
I20260928 23:17:31.232857 8398 ts_tablet_manager.cc:1397] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:31.232959 8398 raft_consensus.cc:348] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260928 23:17:31.233027 8398 raft_consensus.cc:374] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:31.233045 8398 raft_consensus.cc:729] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffc81c6d96c4e97b655efdb67bcee5e, State: Initialized, Role: FOLLOWER
I20260928 23:17:31.233089 8398 consensus_queue.cc:260] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260928 23:17:31.233176 8398 ts_tablet_manager.cc:1428] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:31.233260 8417 tablet_replica.cc:439] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } }
I20260928 23:17:31.233326 8417 tablet_replica.cc:442] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:31.320247 8417 raft_consensus.cc:482] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:31.320339 8417 raft_consensus.cc:504] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260928 23:17:31.320473 8417 leader_election.cc:288] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365), f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401)
I20260928 23:17:31.320683 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ace2b11451648088bac04c82d73d3d8" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e7bad34484c31864d4948fc8d8a51" is_pre_election: true
I20260928 23:17:31.320688 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ace2b11451648088bac04c82d73d3d8" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" is_pre_election: true
I20260928 23:17:31.320770 8043 raft_consensus.cc:2454] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 0.
I20260928 23:17:31.320782 8183 raft_consensus.cc:2454] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 0.
I20260928 23:17:31.320920 8254 leader_election.cc:302] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ffc81c6d96c4e97b655efdb67bcee5e, d79e7bad34484c31864d4948fc8d8a51; no voters:
I20260928 23:17:31.321039 8417 raft_consensus.cc:2781] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:31.321072 8417 raft_consensus.cc:482] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:31.321096 8417 raft_consensus.cc:3037] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.321758 8417 raft_consensus.cc:504] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260928 23:17:31.321888 8417 leader_election.cc:288] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 election: Requested vote from peers d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365), f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401)
I20260928 23:17:31.322002 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ace2b11451648088bac04c82d73d3d8" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e7bad34484c31864d4948fc8d8a51"
I20260928 23:17:31.322053 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ace2b11451648088bac04c82d73d3d8" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3"
I20260928 23:17:31.322089 8043 raft_consensus.cc:3037] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.322124 8183 raft_consensus.cc:3037] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.322806 8183 raft_consensus.cc:2454] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 1.
I20260928 23:17:31.322808 8043 raft_consensus.cc:2454] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 1.
I20260928 23:17:31.323004 8254 leader_election.cc:302] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ffc81c6d96c4e97b655efdb67bcee5e, d79e7bad34484c31864d4948fc8d8a51; no voters:
I20260928 23:17:31.323088 8417 raft_consensus.cc:2781] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:31.323176 8417 raft_consensus.cc:686] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 LEADER]: Becoming Leader. State: Replica: 9ffc81c6d96c4e97b655efdb67bcee5e, State: Running, Role: LEADER
I20260928 23:17:31.323235 8417 consensus_queue.cc:237] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } }
I20260928 23:17:31.323491 8404 tablet_replica.cc:439] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: TxnStatusTablet state changed. Reason: New leader 9ffc81c6d96c4e97b655efdb67bcee5e. Latest consensus state: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } }
I20260928 23:17:31.323582 8404 tablet_replica.cc:442] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:17:31.323728 8437 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:17:31.323840 8437 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:17:31.323817 7886 catalog_manager.cc:5495] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } health_report { overall_health: UNKNOWN } } }
I20260928 23:17:31.349604 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.347201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.0.98.3:33853)
0928 23:17:31.347252 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:31.347254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:31.347261 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:31.347286 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:31.347287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:31.347295 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:31.347338 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:31.347475 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.347990 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.349051 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.349221 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.349266 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.349419 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:31.349442 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:31.349471 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:31.349533 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 23:17:31.349550 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:17:31.349658 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.346957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33853 (local address 127.0.0.1:54936)
0928 23:17:31.347028 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:31.347042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:31.347104 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:31.347366 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:31.347366 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:31.347368 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:17:31.347420 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:31.347422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.348051 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.348052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:31.348979 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:31.348981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.349280 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.349281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:31.349339 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.349354 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:17:31.349482 (+ 128us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:17:31.349487 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:17:31.349531 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:17:31.349578 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 23:17:31.350579 8183 raft_consensus.cc:1261] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ffc81c6d96c4e97b655efdb67bcee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:31.350586 8043 raft_consensus.cc:1261] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Refusing update from remote peer 9ffc81c6d96c4e97b655efdb67bcee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:31.350773 8417 consensus_queue.cc:1035] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:31.350879 8404 consensus_queue.cc:1035] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:31.351243 8403 tablet_replica.cc:439] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: TxnStatusTablet state changed. Reason: New leader 9ffc81c6d96c4e97b655efdb67bcee5e. Latest consensus state: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } }
I20260928 23:17:31.351329 8403 tablet_replica.cc:442] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:31.351394 8405 tablet_replica.cc:439] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: TxnStatusTablet state changed. Reason: New leader 9ffc81c6d96c4e97b655efdb67bcee5e. Latest consensus state: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } }
I20260928 23:17:31.351488 8405 tablet_replica.cc:442] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:31.352092 8410 mvcc.cc:205] Tried to move back new op lower bound from 7334451000730578944 to 7334451000620445696. Current Snapshot: MvccSnapshot[applied={T|T < 7334451000730578944}]
I20260928 23:17:31.352561 8405 tablet_replica.cc:439] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } }
I20260928 23:17:31.352655 8405 tablet_replica.cc:442] T 3ace2b11451648088bac04c82d73d3d8 P d79e7bad34484c31864d4948fc8d8a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:31.352619 8417 tablet_replica.cc:439] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } }
I20260928 23:17:31.352701 8417 tablet_replica.cc:442] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:17:31.352952 8417 tablet_replica.cc:439] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } }
I20260928 23:17:31.353019 8417 tablet_replica.cc:442] T 3ace2b11451648088bac04c82d73d3d8 P 9ffc81c6d96c4e97b655efdb67bcee5e: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:17:31.353404 8412 mvcc.cc:205] Tried to move back new op lower bound from 7334451000730578944 to 7334451000620445696. Current Snapshot: MvccSnapshot[applied={T|T < 7334451000730578944}]
I20260928 23:17:31.353585 8403 tablet_replica.cc:439] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } }
I20260928 23:17:31.353658 8403 tablet_replica.cc:442] T 3ace2b11451648088bac04c82d73d3d8 P f838121d0c3a42c6a46f4074b0e0bcf3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:17:31.362002 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.356458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33853 (local address 127.0.98.3:60855)
0928 23:17:31.359660 (+ 3202us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:31.359670 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:31.359720 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:31.359836 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:31.359836 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:31.359840 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:17:31.359880 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:31.359881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.360692 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.360692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:31.361247 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:31.361248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.361664 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.361665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:31.361707 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.361719 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:17:31.361868 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:17:31.361869 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 23:17:31.361897 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 23:17:31.361929 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3171,"thread_start_us":72,"threads_started":1}
I20260928 23:17:31.362003 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.356515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60855 (local address 127.0.98.3:33853)
0928 23:17:31.356551 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:31.356553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:31.359726 (+ 3173us) server_negotiation.cc:407] Connection header received
0928 23:17:31.359759 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:31.359760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:31.359770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:31.359832 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:31.359915 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.360613 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.361311 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.361601 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.361663 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.361790 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:31.361820 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:31.361866 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:31.361902 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 23:17:31.361922 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:17:31.363884 8183 raft_consensus.cc:1261] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ffc81c6d96c4e97b655efdb67bcee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:31.363922 8043 raft_consensus.cc:1261] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Refusing update from remote peer 9ffc81c6d96c4e97b655efdb67bcee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:31.364092 8404 consensus_queue.cc:1035] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:31.364195 8417 consensus_queue.cc:1035] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:31.373904 7885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:31.374003 7885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:31.374548 7885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08: Sending DeleteTablet for 3 replicas of tablet 5543676df57a40b293fdc94442665c6d
I20260928 23:17:31.374758 8152 tablet_service.cc:1514] Processing DeleteTablet for tablet 5543676df57a40b293fdc94442665c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260928 23:17:31.374792 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 5543676df57a40b293fdc94442665c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54926
I20260928 23:17:31.374850 8420 tablet_replica.cc:330] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3: stopping tablet replica
I20260928 23:17:31.374893 8420 raft_consensus.cc:2229] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:31.374922 8420 raft_consensus.cc:2258] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:31.374943 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 5543676df57a40b293fdc94442665c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50552
I20260928 23:17:31.374979 8422 tablet_replica.cc:330] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: stopping tablet replica
I20260928 23:17:31.375022 8422 raft_consensus.cc:2229] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:31.375065 8422 raft_consensus.cc:2258] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:31.375212 8421 tablet_replica.cc:330] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51: stopping tablet replica
I20260928 23:17:31.375260 8421 raft_consensus.cc:2229] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:31.375281 8421 raft_consensus.cc:2258] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:31.375608 8421 ts_tablet_manager.cc:1905] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:31.376995 8422 ts_tablet_manager.cc:1905] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:31.376986 8420 ts_tablet_manager.cc:1905] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:31.378381 8422 ts_tablet_manager.cc:1918] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:31.378397 8420 ts_tablet_manager.cc:1918] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:31.378439 8422 log.cc:1196] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/5543676df57a40b293fdc94442665c6d
I20260928 23:17:31.378439 8420 log.cc:1196] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/5543676df57a40b293fdc94442665c6d
I20260928 23:17:31.378633 8420 ts_tablet_manager.cc:1939] T 5543676df57a40b293fdc94442665c6d P f838121d0c3a42c6a46f4074b0e0bcf3: Deleting consensus metadata
I20260928 23:17:31.378641 8422 ts_tablet_manager.cc:1939] T 5543676df57a40b293fdc94442665c6d P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting consensus metadata
I20260928 23:17:31.378885 8421 ts_tablet_manager.cc:1918] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:31.378940 8421 log.cc:1196] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/5543676df57a40b293fdc94442665c6d
I20260928 23:17:31.379123 8421 ts_tablet_manager.cc:1939] T 5543676df57a40b293fdc94442665c6d P d79e7bad34484c31864d4948fc8d8a51: Deleting consensus metadata
I20260928 23:17:31.379382 7870 catalog_manager.cc:4841] TS d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365): tablet 5543676df57a40b293fdc94442665c6d (table test-table [id=eb2332461f774a8f8c6959c79b69e4fc]) successfully deleted
I20260928 23:17:31.381727 7871 catalog_manager.cc:4841] TS f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401): tablet 5543676df57a40b293fdc94442665c6d (table test-table [id=eb2332461f774a8f8c6959c79b69e4fc]) successfully deleted
I20260928 23:17:31.381785 7872 catalog_manager.cc:4841] TS 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3:33853): tablet 5543676df57a40b293fdc94442665c6d (table test-table [id=eb2332461f774a8f8c6959c79b69e4fc]) successfully deleted
Sep 28 23:17:31 dist-test-slave-djkr krb5kdc[7841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:17:31 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:31 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:17:31.400364 7954 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:31.394228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.0.98.62:45297)
0928 23:17:31.394291 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:31.394293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:31.394407 (+ 114us) server_negotiation.cc:407] Connection header received
0928 23:17:31.394483 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:31.394484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:31.394493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:31.394540 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:31.395251 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.395803 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.396885 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.397104 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.397168 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.400303 (+ 3135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60394: BlockingRecv error: recv got EOF from 127.0.0.1:60394 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 23:17:31 dist-test-slave-djkr krb5kdc[7841](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:17:31.408313 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.401366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.0.98.62:45297)
0928 23:17:31.401402 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:31.401404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:31.401466 (+ 62us) server_negotiation.cc:407] Connection header received
0928 23:17:31.401534 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:31.401534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:31.401543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:31.401584 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:31.401997 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.402451 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.403076 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.403257 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.403308 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.405724 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:31.405725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:31.405726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:31.405731 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:31.407092 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:31.407507 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:31.407508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:31.407508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:31.407609 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:31.407857 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:31.407858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:31.407859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:31.407965 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:31.408058 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:31.408151 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 23:17:31.408226 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:17:31.409700 7885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:31.409863 7885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:31.411098 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 022a8fff6a1e4f248fe957f4955df3a6 (DEFAULT_TABLE table=test-table [id=9a9f26d8489f4fea91eed5261e390422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:31.411149 8023 tablet_service.cc:1467] Processing CreateTablet for tablet 022a8fff6a1e4f248fe957f4955df3a6 (DEFAULT_TABLE table=test-table [id=9a9f26d8489f4fea91eed5261e390422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:31.411226 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022a8fff6a1e4f248fe957f4955df3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:31.411296 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022a8fff6a1e4f248fe957f4955df3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:31.412562 8152 tablet_service.cc:1467] Processing CreateTablet for tablet 022a8fff6a1e4f248fe957f4955df3a6 (DEFAULT_TABLE table=test-table [id=9a9f26d8489f4fea91eed5261e390422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:31.412673 8152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022a8fff6a1e4f248fe957f4955df3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:31.412781 8399 tablet_bootstrap.cc:492] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51: Bootstrap starting.
I20260928 23:17:31.413170 8399 tablet_bootstrap.cc:654] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:31.413787 8399 tablet_bootstrap.cc:492] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51: No bootstrap required, opened a new log
I20260928 23:17:31.413828 8399 ts_tablet_manager.cc:1397] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:31.413950 8399 raft_consensus.cc:348] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.413983 8398 tablet_bootstrap.cc:492] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e: Bootstrap starting.
I20260928 23:17:31.414011 8399 raft_consensus.cc:374] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:31.414029 8399 raft_consensus.cc:729] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79e7bad34484c31864d4948fc8d8a51, State: Initialized, Role: FOLLOWER
I20260928 23:17:31.414067 8399 consensus_queue.cc:260] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.414139 8399 ts_tablet_manager.cc:1428] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:31.414240 8398 tablet_bootstrap.cc:654] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:31.414534 8397 tablet_bootstrap.cc:492] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3: Bootstrap starting.
I20260928 23:17:31.414901 8397 tablet_bootstrap.cc:654] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:31.415611 8397 tablet_bootstrap.cc:492] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3: No bootstrap required, opened a new log
I20260928 23:17:31.415658 8397 ts_tablet_manager.cc:1397] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:31.415791 8398 tablet_bootstrap.cc:492] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e: No bootstrap required, opened a new log
I20260928 23:17:31.415798 8397 raft_consensus.cc:348] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.415853 8397 raft_consensus.cc:374] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:31.415848 8398 ts_tablet_manager.cc:1397] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 23:17:31.415879 8397 raft_consensus.cc:729] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f838121d0c3a42c6a46f4074b0e0bcf3, State: Initialized, Role: FOLLOWER
I20260928 23:17:31.415920 8397 consensus_queue.cc:260] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.415966 8398 raft_consensus.cc:348] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.416019 8397 ts_tablet_manager.cc:1428] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:31.416024 8398 raft_consensus.cc:374] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:31.416044 8398 raft_consensus.cc:729] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffc81c6d96c4e97b655efdb67bcee5e, State: Initialized, Role: FOLLOWER
I20260928 23:17:31.416078 8398 consensus_queue.cc:260] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.416160 8398 ts_tablet_manager.cc:1428] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:31.640707 8403 raft_consensus.cc:482] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:31.640821 8403 raft_consensus.cc:504] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.641191 8403 leader_election.cc:288] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365), 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3:33853)
I20260928 23:17:31.644125 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.641424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38373 (local address 127.0.98.3:33853)
0928 23:17:31.641470 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:31.641471 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:31.641586 (+ 115us) server_negotiation.cc:407] Connection header received
0928 23:17:31.641663 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:31.641664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:31.641673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:31.641722 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:31.641881 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.642342 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.643425 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.643822 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.643885 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.643897 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:31.644023 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 23:17:31.644059 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:17:31.644145 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.641419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60413 (local address 127.0.98.1:35365)
0928 23:17:31.641514 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:31.641515 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:31.641527 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:17:31.641587 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:31.641587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:31.641598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:31.641659 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:31.641819 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.642358 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.643392 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.643807 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.643860 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.643888 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:31.644020 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 23:17:31.644059 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":42,"threads_started":1}
I20260928 23:17:31.644166 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.641352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33853 (local address 127.0.98.2:38373)
0928 23:17:31.641511 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:31.641522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:31.641588 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:31.641749 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:31.641750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:31.641751 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:31.641809 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:31.641811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.642418 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.642418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:31.643323 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:31.643333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.643901 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.643902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:31.643963 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.643966 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:31.644013 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:17:31.644053 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":103,"thread_start_us":62,"threads_started":1}
I20260928 23:17:31.644172 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.641351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35365 (local address 127.0.98.2:60413)
0928 23:17:31.641473 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:31.641484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:31.641535 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:31.641666 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:31.641666 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:31.641674 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:31.641754 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:31.641756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.642455 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.642456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:31.643291 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:31.643293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.643885 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.643886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:31.643952 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.643956 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:31.644009 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:17:31.644053 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":30,"thread_start_us":68,"threads_started":1}
I20260928 23:17:31.644414 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022a8fff6a1e4f248fe957f4955df3a6" candidate_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" is_pre_election: true
I20260928 23:17:31.644495 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022a8fff6a1e4f248fe957f4955df3a6" candidate_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e7bad34484c31864d4948fc8d8a51" is_pre_election: true
I20260928 23:17:31.644555 8323 raft_consensus.cc:2454] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f838121d0c3a42c6a46f4074b0e0bcf3 in term 0.
I20260928 23:17:31.644573 8043 raft_consensus.cc:2454] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f838121d0c3a42c6a46f4074b0e0bcf3 in term 0.
I20260928 23:17:31.644745 8114 leader_election.cc:302] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9ffc81c6d96c4e97b655efdb67bcee5e, d79e7bad34484c31864d4948fc8d8a51, f838121d0c3a42c6a46f4074b0e0bcf3; no voters:
I20260928 23:17:31.644876 8403 raft_consensus.cc:2781] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:31.644920 8403 raft_consensus.cc:482] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:31.644938 8403 raft_consensus.cc:3037] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.645643 8403 raft_consensus.cc:504] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.645771 8403 leader_election.cc:288] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [CANDIDATE]: Term 1 election: Requested vote from peers d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365), 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3:33853)
I20260928 23:17:31.645911 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022a8fff6a1e4f248fe957f4955df3a6" candidate_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e7bad34484c31864d4948fc8d8a51"
I20260928 23:17:31.645951 8323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022a8fff6a1e4f248fe957f4955df3a6" candidate_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e"
I20260928 23:17:31.645987 8043 raft_consensus.cc:3037] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.646014 8323 raft_consensus.cc:3037] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.646603 8323 raft_consensus.cc:2454] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f838121d0c3a42c6a46f4074b0e0bcf3 in term 1.
I20260928 23:17:31.646768 8116 leader_election.cc:302] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ffc81c6d96c4e97b655efdb67bcee5e, f838121d0c3a42c6a46f4074b0e0bcf3; no voters:
I20260928 23:17:31.646891 8043 raft_consensus.cc:2454] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f838121d0c3a42c6a46f4074b0e0bcf3 in term 1.
I20260928 23:17:31.646885 8403 raft_consensus.cc:2781] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:31.647012 8403 raft_consensus.cc:686] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 LEADER]: Becoming Leader. State: Replica: f838121d0c3a42c6a46f4074b0e0bcf3, State: Running, Role: LEADER
I20260928 23:17:31.647084 8403 consensus_queue.cc:237] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.647845 7886 catalog_manager.cc:5495] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } health_report { overall_health: UNKNOWN } } }
I20260928 23:17:31.700743 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.698269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.0.98.2:44401)
0928 23:17:31.698415 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:31.698417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:31.698424 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:17:31.698462 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:31.698463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:31.698481 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:31.698555 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:31.698664 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.699162 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.699920 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.700171 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.700221 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.700383 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:31.700499 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:31.700561 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:31.700617 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:17:31.700637 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260928 23:17:31.701727 8323 raft_consensus.cc:1261] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Refusing update from remote peer f838121d0c3a42c6a46f4074b0e0bcf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:31.701719 8043 raft_consensus.cc:1261] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Refusing update from remote peer f838121d0c3a42c6a46f4074b0e0bcf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:31.701958 8403 consensus_queue.cc:1035] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:31.702096 8467 consensus_queue.cc:1035] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:31.705145 7886 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:31.705224 7886 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:31.705752 7886 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08: Sending DeleteTablet for 3 replicas of tablet 022a8fff6a1e4f248fe957f4955df3a6
I20260928 23:17:31.705973 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet 022a8fff6a1e4f248fe957f4955df3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50552
I20260928 23:17:31.705992 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 022a8fff6a1e4f248fe957f4955df3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54926
I20260928 23:17:31.706082 8421 tablet_replica.cc:330] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51: stopping tablet replica
I20260928 23:17:31.706140 8421 raft_consensus.cc:2229] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:31.706183 8421 raft_consensus.cc:2258] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:31.706250 8422 tablet_replica.cc:330] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e: stopping tablet replica
I20260928 23:17:31.706303 8422 raft_consensus.cc:2229] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:31.706353 8422 raft_consensus.cc:2258] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:31.706593 8421 ts_tablet_manager.cc:1905] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:31.706734 8152 tablet_service.cc:1514] Processing DeleteTablet for tablet 022a8fff6a1e4f248fe957f4955df3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260928 23:17:31.706811 8420 tablet_replica.cc:330] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3: stopping tablet replica
I20260928 23:17:31.706853 8420 raft_consensus.cc:2229] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:31.706929 8420 raft_consensus.cc:2258] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:31.707010 7886 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:31.707135 7886 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:31.707206 8420 ts_tablet_manager.cc:1905] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:31.708326 8152 tablet_service.cc:1467] Processing CreateTablet for tablet b44f2c29720447efacad603fe0d819db (DEFAULT_TABLE table=test-table [id=e77a02e5352041e597e6e16c8a2cb157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:31.708424 8152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44f2c29720447efacad603fe0d819db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:31.708636 8420 ts_tablet_manager.cc:1918] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:31.708684 8420 log.cc:1196] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/022a8fff6a1e4f248fe957f4955df3a6
I20260928 23:17:31.708877 8420 ts_tablet_manager.cc:1939] T 022a8fff6a1e4f248fe957f4955df3a6 P f838121d0c3a42c6a46f4074b0e0bcf3: Deleting consensus metadata
I20260928 23:17:31.709132 8023 tablet_service.cc:1467] Processing CreateTablet for tablet b44f2c29720447efacad603fe0d819db (DEFAULT_TABLE table=test-table [id=e77a02e5352041e597e6e16c8a2cb157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:31.709147 8397 tablet_bootstrap.cc:492] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3: Bootstrap starting.
I20260928 23:17:31.709419 8421 ts_tablet_manager.cc:1918] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:31.709465 8421 log.cc:1196] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/022a8fff6a1e4f248fe957f4955df3a6
I20260928 23:17:31.709517 8397 tablet_bootstrap.cc:654] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:31.709558 8023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44f2c29720447efacad603fe0d819db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:31.709653 8421 ts_tablet_manager.cc:1939] T 022a8fff6a1e4f248fe957f4955df3a6 P d79e7bad34484c31864d4948fc8d8a51: Deleting consensus metadata
I20260928 23:17:31.709915 8303 tablet_service.cc:1467] Processing CreateTablet for tablet b44f2c29720447efacad603fe0d819db (DEFAULT_TABLE table=test-table [id=e77a02e5352041e597e6e16c8a2cb157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:31.710011 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44f2c29720447efacad603fe0d819db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:31.710338 8397 tablet_bootstrap.cc:492] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3: No bootstrap required, opened a new log
I20260928 23:17:31.710373 8397 ts_tablet_manager.cc:1397] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:31.710497 8397 raft_consensus.cc:348] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.710548 8397 raft_consensus.cc:374] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:31.710566 8397 raft_consensus.cc:729] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f838121d0c3a42c6a46f4074b0e0bcf3, State: Initialized, Role: FOLLOWER
I20260928 23:17:31.710603 8397 consensus_queue.cc:260] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.710641 8399 tablet_bootstrap.cc:492] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51: Bootstrap starting.
I20260928 23:17:31.710677 8397 ts_tablet_manager.cc:1428] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:31.710986 8399 tablet_bootstrap.cc:654] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:31.711179 8398 tablet_bootstrap.cc:492] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e: Bootstrap starting.
I20260928 23:17:31.711349 7870 catalog_manager.cc:4841] TS d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365): tablet 022a8fff6a1e4f248fe957f4955df3a6 (table test-table [id=9a9f26d8489f4fea91eed5261e390422]) successfully deleted
I20260928 23:17:31.711449 8398 tablet_bootstrap.cc:654] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:31.711846 7871 catalog_manager.cc:4841] TS f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401): tablet 022a8fff6a1e4f248fe957f4955df3a6 (table test-table [id=9a9f26d8489f4fea91eed5261e390422]) successfully deleted
I20260928 23:17:31.712080 8422 ts_tablet_manager.cc:1905] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:31.712285 8398 tablet_bootstrap.cc:492] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e: No bootstrap required, opened a new log
I20260928 23:17:31.712318 8398 ts_tablet_manager.cc:1397] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:31.712405 8398 raft_consensus.cc:348] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.712457 8398 raft_consensus.cc:374] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:31.712474 8398 raft_consensus.cc:729] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffc81c6d96c4e97b655efdb67bcee5e, State: Initialized, Role: FOLLOWER
I20260928 23:17:31.712510 8398 consensus_queue.cc:260] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.712584 8398 ts_tablet_manager.cc:1428] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:31.712801 8399 tablet_bootstrap.cc:492] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51: No bootstrap required, opened a new log
I20260928 23:17:31.712852 8399 ts_tablet_manager.cc:1397] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:17:31.712990 8399 raft_consensus.cc:348] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.713056 8399 raft_consensus.cc:374] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:31.713076 8399 raft_consensus.cc:729] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79e7bad34484c31864d4948fc8d8a51, State: Initialized, Role: FOLLOWER
I20260928 23:17:31.713112 8399 consensus_queue.cc:260] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.713192 8399 ts_tablet_manager.cc:1428] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:31.713295 8422 ts_tablet_manager.cc:1918] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:31.713346 8422 log.cc:1196] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/022a8fff6a1e4f248fe957f4955df3a6
I20260928 23:17:31.713521 8422 ts_tablet_manager.cc:1939] T 022a8fff6a1e4f248fe957f4955df3a6 P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting consensus metadata
I20260928 23:17:31.713835 7872 catalog_manager.cc:4841] TS 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3:33853): tablet 022a8fff6a1e4f248fe957f4955df3a6 (table test-table [id=9a9f26d8489f4fea91eed5261e390422]) successfully deleted
I20260928 23:17:31.982470 8448 raft_consensus.cc:482] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:31.982573 8448 raft_consensus.cc:504] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.982734 8448 leader_election.cc:288] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365), f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401)
I20260928 23:17:31.982969 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44f2c29720447efacad603fe0d819db" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" is_pre_election: true
I20260928 23:17:31.982968 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44f2c29720447efacad603fe0d819db" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e7bad34484c31864d4948fc8d8a51" is_pre_election: true
I20260928 23:17:31.983062 8043 raft_consensus.cc:2454] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 0.
I20260928 23:17:31.983058 8183 raft_consensus.cc:2454] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 0.
I20260928 23:17:31.983201 8255 leader_election.cc:302] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ffc81c6d96c4e97b655efdb67bcee5e, f838121d0c3a42c6a46f4074b0e0bcf3; no voters:
I20260928 23:17:31.983336 8448 raft_consensus.cc:2781] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:31.983385 8448 raft_consensus.cc:482] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:31.983399 8448 raft_consensus.cc:3037] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.984089 8448 raft_consensus.cc:504] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.984210 8448 leader_election.cc:288] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 election: Requested vote from peers d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365), f838121d0c3a42c6a46f4074b0e0bcf3 (127.0.98.2:44401)
I20260928 23:17:31.984344 8183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44f2c29720447efacad603fe0d819db" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3"
I20260928 23:17:31.984416 8183 raft_consensus.cc:3037] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.984391 8043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44f2c29720447efacad603fe0d819db" candidate_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e7bad34484c31864d4948fc8d8a51"
I20260928 23:17:31.984455 8043 raft_consensus.cc:3037] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:31.984985 8183 raft_consensus.cc:2454] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 1.
I20260928 23:17:31.984994 8043 raft_consensus.cc:2454] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ffc81c6d96c4e97b655efdb67bcee5e in term 1.
I20260928 23:17:31.985154 8255 leader_election.cc:302] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9ffc81c6d96c4e97b655efdb67bcee5e, d79e7bad34484c31864d4948fc8d8a51, f838121d0c3a42c6a46f4074b0e0bcf3; no voters:
I20260928 23:17:31.985252 8448 raft_consensus.cc:2781] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:31.985369 8448 raft_consensus.cc:686] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 LEADER]: Becoming Leader. State: Replica: 9ffc81c6d96c4e97b655efdb67bcee5e, State: Running, Role: LEADER
I20260928 23:17:31.985421 8448 consensus_queue.cc:237] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } }
I20260928 23:17:31.985952 7886 catalog_manager.cc:5495] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ffc81c6d96c4e97b655efdb67bcee5e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33853 } health_report { overall_health: HEALTHY } } }
I20260928 23:17:31.994068 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:31.992059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.0.98.3:33853)
0928 23:17:31.992100 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:31.992101 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:31.992111 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:31.992170 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:31.992171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:31.992180 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:31.992229 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:31.992325 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.992790 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.993488 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:31.993686 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:31.993738 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:31.993866 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:31.993901 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:31.993940 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:31.993988 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:17:31.994006 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:17:31.995898 8043 raft_consensus.cc:1261] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Refusing update from remote peer 9ffc81c6d96c4e97b655efdb67bcee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:31.995915 8183 raft_consensus.cc:1261] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Refusing update from remote peer 9ffc81c6d96c4e97b655efdb67bcee5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:31.996167 8448 consensus_queue.cc:1035] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f838121d0c3a42c6a46f4074b0e0bcf3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:31.996279 8480 consensus_queue.cc:1035] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d79e7bad34484c31864d4948fc8d8a51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:32.004997 7885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:32.005077 7885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:32.005575 7885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 163b0b9acfa5432fb81babb620610e08: Sending DeleteTablet for 3 replicas of tablet b44f2c29720447efacad603fe0d819db
I20260928 23:17:32.005784 8023 tablet_service.cc:1514] Processing DeleteTablet for tablet b44f2c29720447efacad603fe0d819db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50552
I20260928 23:17:32.005815 8152 tablet_service.cc:1514] Processing DeleteTablet for tablet b44f2c29720447efacad603fe0d819db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260928 23:17:32.005872 8302 tablet_service.cc:1514] Processing DeleteTablet for tablet b44f2c29720447efacad603fe0d819db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54926
I20260928 23:17:32.005911 8421 tablet_replica.cc:330] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51: stopping tablet replica
I20260928 23:17:32.005906 8420 tablet_replica.cc:330] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3: stopping tablet replica
I20260928 23:17:32.005966 8421 raft_consensus.cc:2229] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:32.005966 8420 raft_consensus.cc:2229] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:32.005997 8421 raft_consensus.cc:2258] T b44f2c29720447efacad603fe0d819db P d79e7bad34484c31864d4948fc8d8a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:32.006000 8420 raft_consensus.cc:2258] T b44f2c29720447efacad603fe0d819db P f838121d0c3a42c6a46f4074b0e0bcf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:32.006124 8422 tablet_replica.cc:330] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e: stopping tablet replica
I20260928 23:17:32.006177 8422 raft_consensus.cc:2229] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:32.006251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 7958
I20260928 23:17:32.006282 8422 raft_consensus.cc:2258] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 23:17:32.010896 7870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35365 (error 108)
I20260928 23:17:32.011057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 8098
I20260928 23:17:32.011991 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:32.011202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35365 (local address 127.0.0.1:50562)
0928 23:17:32.011911 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:32.011956 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":44,"threads_started":1}
I20260928 23:17:32.011991 8422 ts_tablet_manager.cc:1905] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:17:32.012344 7870 catalog_manager.cc:4584] TS d79e7bad34484c31864d4948fc8d8a51 (127.0.98.1:35365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b44f2c29720447efacad603fe0d819db: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35365: connect: Connection refused (error 111)
I20260928 23:17:32.013556 8422 ts_tablet_manager.cc:1918] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:17:32.013610 8422 log.cc:1196] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/b44f2c29720447efacad603fe0d819db
I20260928 23:17:32.013784 8422 ts_tablet_manager.cc:1939] T b44f2c29720447efacad603fe0d819db P 9ffc81c6d96c4e97b655efdb67bcee5e: Deleting consensus metadata
I20260928 23:17:32.014104 7872 catalog_manager.cc:4841] TS 9ffc81c6d96c4e97b655efdb67bcee5e (127.0.98.3:33853): tablet b44f2c29720447efacad603fe0d819db (table test-table [id=e77a02e5352041e597e6e16c8a2cb157]) successfully deleted
I20260928 23:17:32.016362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 8238
I20260928 23:17:32.017062 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:32.016357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44401 (local address 127.0.0.1:54872)
0928 23:17:32.016404 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:32.017034 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 23:17:32.020751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 7854
2026-09-28T23:17:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2971 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:32 dist-test-slave-djkr krb5kdc[8489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:32 dist-test-slave-djkr krb5kdc[8489](info): set up 2 sockets
Sep 28 23:17:32 dist-test-slave-djkr krb5kdc[8489](info): commencing operation
krb5kdc: starting...
W20260928 23:17:32.975781 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:33 dist-test-slave-djkr krb5kdc[8489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:33.086035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:33.162858 8502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:33.163041 8502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:33.163061 8502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:33.164227 8502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:33.164268 8502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:33.164281 8502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:33.164292 8502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:33.164302 8502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:33.165773 8502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:33.165982 8502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:33.166183 8502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:33.170608 8508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:33.170620 8510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:33.170636 8502 server_base.cc:1034] running on GCE node
W20260928 23:17:33.170620 8507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:33.171100 8502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:33.171393 8502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:33.172541 8502 hybrid_clock.cc:648] HybridClock initialized: now 1790637453172516 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T23:17:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1153 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:33 dist-test-slave-djkr krb5kdc[8515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:33 dist-test-slave-djkr krb5kdc[8515](info): set up 2 sockets
Sep 28 23:17:33 dist-test-slave-djkr krb5kdc[8515](info): commencing operation
krb5kdc: starting...
W20260928 23:17:34.119673 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:34 dist-test-slave-djkr krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:34.227046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42319
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:34.302944 8528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:34.303139 8528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:34.303160 8528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:34.304330 8528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:34.304371 8528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:34.304384 8528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:34.304395 8528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:34.304406 8528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:34.305917 8528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:34.306139 8528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:34.306336 8528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:34.310740 8533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:34.310737 8534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:34.310762 8536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:34.311005 8528 server_base.cc:1034] running on GCE node
I20260928 23:17:34.311314 8528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:34.311573 8528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:34.312713 8528 hybrid_clock.cc:648] HybridClock initialized: now 1790637454312703 us; error 27 us; skew 500 ppm
Sep 28 23:17:34 dist-test-slave-djkr krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:34.319090 8528 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:34.319455 8528 webserver.cc:460] Webserver started at http://127.0.98.62:39325/ using document root <none> and password file <none>
I20260928 23:17:34.319708 8528 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:34.319784 8528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:34.319880 8528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:34.320847 8528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "5c480826a345432c88a500716f5d164d"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "eae6aaba9a206a80b53f668ed5ed9cc1"
server_key_iv: "b76ab47ad5c1b55f2c9de2bcc61fc7f9"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.321121 8528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c480826a345432c88a500716f5d164d"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "eae6aaba9a206a80b53f668ed5ed9cc1"
server_key_iv: "b76ab47ad5c1b55f2c9de2bcc61fc7f9"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.322223 8528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:34.322804 8543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:34.322945 8528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:34.323016 8528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "5c480826a345432c88a500716f5d164d"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "eae6aaba9a206a80b53f668ed5ed9cc1"
server_key_iv: "b76ab47ad5c1b55f2c9de2bcc61fc7f9"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.323077 8528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:34.333109 8528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:34.334954 8528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:34.335073 8528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:34.338547 8528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42319
I20260928 23:17:34.338590 8595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42319 every 8 connection(s)
I20260928 23:17:34.338999 8528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:34.339645 8596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:34.340747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 8528
I20260928 23:17:34.340849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:34.341019 392 external_mini_cluster.cc:1428] Setting key c0cc8090b00a40aa9f154ca4ffc7b6eb
I20260928 23:17:34.342056 8596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d: Bootstrap starting.
I20260928 23:17:34.342563 8596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:34.342808 8596 log.cc:824] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:34.343356 8596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d: No bootstrap required, opened a new log
I20260928 23:17:34.345055 8596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
I20260928 23:17:34.345198 8596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:34.345230 8596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c480826a345432c88a500716f5d164d, State: Initialized, Role: FOLLOWER
I20260928 23:17:34.345366 8596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
I20260928 23:17:34.345443 8596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:34.345479 8596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:34.345525 8596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:34.346145 8596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
I20260928 23:17:34.346247 8596 leader_election.cc:302] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c480826a345432c88a500716f5d164d; no voters:
I20260928 23:17:34.346401 8596 leader_election.cc:288] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:34.346444 8601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:34.346581 8601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 1 LEADER]: Becoming Leader. State: Replica: 5c480826a345432c88a500716f5d164d, State: Running, Role: LEADER
I20260928 23:17:34.346640 8596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:34.346661 8601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
Sep 28 23:17:34 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:34.346990 8601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c480826a345432c88a500716f5d164d. Latest consensus state: current_term: 1 leader_uuid: "5c480826a345432c88a500716f5d164d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } } }
I20260928 23:17:34.346997 8603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c480826a345432c88a500716f5d164d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } } }
I20260928 23:17:34.347059 8601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:34.347097 8603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:34.347318 8607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:34.348312 8607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:34.349670 8607 catalog_manager.cc:1347] Generated new cluster ID: 0651fa6156b24ed79af2e0cad1cc322c
I20260928 23:17:34.349716 8607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:34.350878 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:34.341485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.0.98.62:42319)
0928 23:17:34.341648 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:34.341655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:34.341678 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:17:34.342047 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:34.342053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:34.342096 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:34.342223 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:34.342805 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.343798 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.344501 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.344897 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.344986 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:34.347581 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:34.347585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:34.347585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:34.347596 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:34.349340 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:34.349737 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:34.349739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:34.349740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:34.349835 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:34.350033 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:34.350034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:34.350035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:34.350311 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:34.350438 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:34.350573 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 23:17:34.350689 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260928 23:17:34.381219 8607 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:34.381722 8607 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:34.384639 8607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d: Generated new TSK 0
I20260928 23:17:34.384805 8607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:17:34.417855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:34.513849 8622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:34.514046 8622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:34.514067 8622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:34.515173 8622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:34.515213 8622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:34.515236 8622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:34.516706 8622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:34.516891 8622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:34.517056 8622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:17:34.521730 8622 server_base.cc:1034] running on GCE node
W20260928 23:17:34.521651 8627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:34.521651 8628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:34.521654 8630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:34.522070 8622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:34.522297 8622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:34.523428 8622 hybrid_clock.cc:648] HybridClock initialized: now 1790637454523416 us; error 30 us; skew 500 ppm
Sep 28 23:17:34 dist-test-slave-djkr krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:34.525913 8622 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:17:34.526358 8622 webserver.cc:460] Webserver started at http://127.0.98.1:45371/ using document root <none> and password file <none>
I20260928 23:17:34.526592 8622 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:34.526639 8622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:34.526786 8622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:34.527822 8622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "2616aeb287c54c269fca354925654a11"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "3aa472465266604f634578816a88a840"
server_key_iv: "c7c13585e790fb3631ed089ec6abec7f"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.528131 8622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2616aeb287c54c269fca354925654a11"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "3aa472465266604f634578816a88a840"
server_key_iv: "c7c13585e790fb3631ed089ec6abec7f"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.529266 8622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:34.529883 8637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:34.530020 8622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:34.530095 8622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "2616aeb287c54c269fca354925654a11"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "3aa472465266604f634578816a88a840"
server_key_iv: "c7c13585e790fb3631ed089ec6abec7f"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.530158 8622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:34.537707 8622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:34.539609 8622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:34.539729 8622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:34.539912 8622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:34.540167 8622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:34.540197 8622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:34.540221 8622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:34.540244 8622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:34.545387 8622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32999
I20260928 23:17:34.545439 8750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32999 every 8 connection(s)
I20260928 23:17:34.545689 8622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:17:34 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:34.552186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 8622
I20260928 23:17:34.552279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:34.552438 392 external_mini_cluster.cc:1428] Setting key 108e586c784c4a65496f52ab40a2826a
I20260928 23:17:34.554557 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:34.546352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37117 (local address 127.0.98.62:42319)
0928 23:17:34.546404 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:34.546405 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:34.546603 (+ 198us) server_negotiation.cc:407] Connection header received
0928 23:17:34.547529 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:34.547530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:34.547540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:34.547631 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:34.548181 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.548712 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.549616 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.549796 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.549843 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:34.551725 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:34.551726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:34.551726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:34.551732 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:34.553419 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:34.553747 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:34.553748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:34.553749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:34.553798 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:34.553997 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:34.553998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:34.553999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:34.554095 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:34.554147 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:34.554395 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 23:17:34.554474 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:17:34.554702 8753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:34.546359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:37117)
0928 23:17:34.546497 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:34.546544 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:34.547454 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:34.547704 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:34.547708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:34.547944 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:34.548119 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:34.548123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.548775 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.548776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:34.549557 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:34.549559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.549847 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.549848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:34.549934 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:34.550350 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:34.550358 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:34.551642 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:34.553485 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:34.553491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:34.553493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:34.553688 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:34.553849 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:34.553849 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:34.553850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:34.553946 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:34.554146 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:34.554152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:34.554277 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 23:17:34.554368 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 23:17:34.554506 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260928 23:17:34.555109 8751 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42319
I20260928 23:17:34.555189 8751 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:34.555343 8751 heartbeater.cc:511] Master 127.0.98.62:42319 requested a full tablet report, sending...
I20260928 23:17:34.555796 8560 ts_manager.cc:194] Registered new tserver with Master: 2616aeb287c54c269fca354925654a11 (127.0.98.1:32999)
I20260928 23:17:34.556365 8560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37117
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:17:34.574240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:34.652328 8756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:34.652509 8756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:34.652530 8756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:34.653601 8756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:34.653635 8756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:34.653657 8756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:34.655038 8756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:34.655211 8756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:34.655375 8756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:34.659977 8764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:34.659971 8761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:34.659991 8762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:34.660144 8756 server_base.cc:1034] running on GCE node
I20260928 23:17:34.660356 8756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:34.660567 8756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:34.661703 8756 hybrid_clock.cc:648] HybridClock initialized: now 1790637454661699 us; error 39 us; skew 500 ppm
Sep 28 23:17:34 dist-test-slave-djkr krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:34.664062 8756 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:17:34.664451 8756 webserver.cc:460] Webserver started at http://127.0.98.2:37799/ using document root <none> and password file <none>
I20260928 23:17:34.664616 8756 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:34.664654 8756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:34.664753 8756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:34.665721 8756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "c5a294a373c34f6d933fb39a63b15e24"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "52162d0022871d6206782f54ad3f4fd5"
server_key_iv: "77a8eaa116bcbacd048da7530b5b9aa4"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.666010 8756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c5a294a373c34f6d933fb39a63b15e24"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "52162d0022871d6206782f54ad3f4fd5"
server_key_iv: "77a8eaa116bcbacd048da7530b5b9aa4"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.667151 8756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:17:34.667882 8771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:34.668042 8756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 23:17:34.668154 8756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "c5a294a373c34f6d933fb39a63b15e24"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "52162d0022871d6206782f54ad3f4fd5"
server_key_iv: "77a8eaa116bcbacd048da7530b5b9aa4"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.668212 8756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:34.678040 8756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:34.679673 8756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:34.679775 8756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:34.679939 8756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:34.680166 8756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:34.680192 8756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:34.680218 8756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:34.680235 8756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:34.685228 8756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38815
I20260928 23:17:34.685295 8884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38815 every 8 connection(s)
I20260928 23:17:34.685547 8756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:34.688210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 8756
I20260928 23:17:34.688305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:34.688462 392 external_mini_cluster.cc:1428] Setting key 783c072a08ad37482c52057e871565ff
Sep 28 23:17:34 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:34.694442 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:34.686168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43839 (local address 127.0.98.62:42319)
0928 23:17:34.686217 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:34.686219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:34.686400 (+ 181us) server_negotiation.cc:407] Connection header received
0928 23:17:34.687306 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:34.687307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:34.687318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:34.687368 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:34.688000 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.688609 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.689666 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.689863 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.689916 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:34.692006 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:34.692007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:34.692007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:34.692016 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:34.693271 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:34.693621 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:34.693622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:34.693623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:34.693679 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:34.693881 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:34.693882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:34.693882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:34.693982 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:34.694036 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:34.694285 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 23:17:34.694359 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:17:34.694643 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:34.686132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:43839)
0928 23:17:34.686289 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:34.686324 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:34.687204 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:34.687438 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:34.687442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:34.687737 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:34.687914 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:34.687918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.688688 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.688689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:34.689582 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:34.689585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.689927 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.689928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:34.690050 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:34.690472 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:34.690480 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:34.691898 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:34.693346 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:34.693352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:34.693354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:34.693558 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:34.693731 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:34.693732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:34.693732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:34.693830 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:34.694032 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:34.694037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:34.694174 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 23:17:34.694257 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:17:34.694412 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260928 23:17:34.695053 8885 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42319
I20260928 23:17:34.695173 8885 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:34.695358 8885 heartbeater.cc:511] Master 127.0.98.62:42319 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:17:34.695716 8560 ts_manager.cc:194] Registered new tserver with Master: c5a294a373c34f6d933fb39a63b15e24 (127.0.98.2:38815)
I20260928 23:17:34.696070 8560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:17:34.710223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:34.787482 8890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:34.787729 8890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:34.787752 8890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:34.788861 8890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:34.788893 8890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:34.788918 8890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:34.790336 8890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:34.790516 8890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:34.790683 8890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:34.795254 8896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:34.795249 8895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:34.795249 8898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:34.795323 8890 server_base.cc:1034] running on GCE node
I20260928 23:17:34.795644 8890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:34.795841 8890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:34.796978 8890 hybrid_clock.cc:648] HybridClock initialized: now 1790637454796969 us; error 25 us; skew 500 ppm
Sep 28 23:17:34 dist-test-slave-djkr krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:34.799223 8890 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:17:34.799681 8890 webserver.cc:460] Webserver started at http://127.0.98.3:42807/ using document root <none> and password file <none>
I20260928 23:17:34.799880 8890 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:34.799922 8890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:34.800038 8890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:34.801009 8890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "2af72b52bfaf4e2196759774c66f715f"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "20ab26228e78f0e04c01507a3419449b"
server_key_iv: "542852c2673b18138389c5c63f450417"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.801342 8890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2af72b52bfaf4e2196759774c66f715f"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "20ab26228e78f0e04c01507a3419449b"
server_key_iv: "542852c2673b18138389c5c63f450417"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.802528 8890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:17:34.803119 8905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:34.803268 8890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:34.803349 8890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "2af72b52bfaf4e2196759774c66f715f"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "20ab26228e78f0e04c01507a3419449b"
server_key_iv: "542852c2673b18138389c5c63f450417"
server_key_version: "encryptionkey@0"
I20260928 23:17:34.803400 8890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:34.813580 8890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:34.815425 8890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:34.815526 8890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:34.815732 8890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:34.815966 8890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:34.815999 8890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:34.816021 8890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:34.816035 8890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:34.821076 8890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45267
I20260928 23:17:34.821115 9018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45267 every 8 connection(s)
I20260928 23:17:34.821390 8890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:34.824012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 8890
I20260928 23:17:34.824097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:34.824256 392 external_mini_cluster.cc:1428] Setting key 0a810c08a452daca662b7a501e336eb1
Sep 28 23:17:34 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:34.829764 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:34.821990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34119 (local address 127.0.98.62:42319)
0928 23:17:34.822060 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:34.822062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:34.822205 (+ 143us) server_negotiation.cc:407] Connection header received
0928 23:17:34.823044 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:34.823045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:34.823054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:34.823104 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:34.823661 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.824209 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.825428 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.825650 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.825700 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:34.827702 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:34.827704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:34.827704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:34.827709 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:34.828728 (+ 1019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:34.829089 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:34.829090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:34.829090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:34.829137 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:34.829327 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:34.829327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:34.829327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:34.829418 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:34.829461 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:34.829650 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 23:17:34.829703 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:17:34.829939 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:34.821960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:34119)
0928 23:17:34.822093 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:34.822126 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:34.822970 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:34.823164 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:34.823167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:34.823387 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:34.823595 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:34.823601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.824284 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.824286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:34.825358 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:34.825360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:34.825735 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:34.825736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:34.825843 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:34.826323 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:34.826331 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:34.827635 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:34.828804 (+ 1169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:34.828809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:34.828811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:34.829026 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:34.829188 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:34.829189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:34.829189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:34.829281 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:34.829458 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:34.829461 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:34.829562 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 23:17:34.829635 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:17:34.829747 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260928 23:17:34.830384 9019 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42319
I20260928 23:17:34.830467 9019 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:34.830626 9019 heartbeater.cc:511] Master 127.0.98.62:42319 requested a full tablet report, sending...
I20260928 23:17:34.830940 8560 ts_manager.cc:194] Registered new tserver with Master: 2af72b52bfaf4e2196759774c66f715f (127.0.98.3:45267)
I20260928 23:17:34.831254 8560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34119
I20260928 23:17:34.835621 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 23:17:34.895059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 8622
I20260928 23:17:34.898732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 8756
I20260928 23:17:34.902331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 8890
I20260928 23:17:34.905673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 8528
I20260928 23:17:34.909654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42319
--webserver_interface=127.0.98.62
--webserver_port=39325
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:34.986299 9028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:34.986508 9028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:34.986531 9028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:34.987704 9028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:34.987746 9028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:34.987758 9028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:34.987769 9028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:34.987780 9028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:34.989284 9028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:34.989516 9028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:34.989718 9028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:34.994172 9036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:34.994168 9034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:34.994292 9033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:34.994465 9028 server_base.cc:1034] running on GCE node
I20260928 23:17:34.994683 9028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:34.994949 9028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:34.996083 9028 hybrid_clock.cc:648] HybridClock initialized: now 1790637454996068 us; error 31 us; skew 500 ppm
Sep 28 23:17:34 dist-test-slave-djkr krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:34.998395 9028 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:34.998826 9028 webserver.cc:460] Webserver started at http://127.0.98.62:39325/ using document root <none> and password file <none>
I20260928 23:17:34.999022 9028 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:34.999082 9028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:35.000432 9028 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:35.001049 9043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.001212 9028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:35.001291 9028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "5c480826a345432c88a500716f5d164d"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "eae6aaba9a206a80b53f668ed5ed9cc1"
server_key_iv: "b76ab47ad5c1b55f2c9de2bcc61fc7f9"
server_key_version: "encryptionkey@0"
I20260928 23:17:35.001650 9028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:35.031468 9028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:35.033313 9028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:35.033450 9028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:35.037055 9028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42319
I20260928 23:17:35.037094 9095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42319 every 8 connection(s)
I20260928 23:17:35.037581 9028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:35.038811 9096 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:17:35.039451 9096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d: Bootstrap starting.
I20260928 23:17:35.044032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 9028
I20260928 23:17:35.044275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32999
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42319
--webserver_port=45371
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:17:35.047737 9096 log.cc:824] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:35.049243 9096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:17:35.049481 9096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d: Bootstrap complete.
I20260928 23:17:35.051477 9096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
I20260928 23:17:35.051757 9096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c480826a345432c88a500716f5d164d, State: Initialized, Role: FOLLOWER
I20260928 23:17:35.051918 9096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
I20260928 23:17:35.052000 9096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:35.052045 9096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:35.052090 9096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:17:35.053006 9096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
I20260928 23:17:35.053099 9096 leader_election.cc:302] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c480826a345432c88a500716f5d164d; no voters:
I20260928 23:17:35.053260 9096 leader_election.cc:288] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:17:35.053319 9100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:17:35.053496 9096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:35.053503 9100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [term 2 LEADER]: Becoming Leader. State: Replica: 5c480826a345432c88a500716f5d164d, State: Running, Role: LEADER
I20260928 23:17:35.053611 9100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
I20260928 23:17:35.053851 9100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c480826a345432c88a500716f5d164d. Latest consensus state: current_term: 2 leader_uuid: "5c480826a345432c88a500716f5d164d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } } }
I20260928 23:17:35.053911 9100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:35.054046 9101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5c480826a345432c88a500716f5d164d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c480826a345432c88a500716f5d164d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } } }
I20260928 23:17:35.054095 9101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:35.054364 9106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:35.054805 9106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:35.054929 9106 catalog_manager.cc:1259] Loaded cluster ID: 0651fa6156b24ed79af2e0cad1cc322c
I20260928 23:17:35.055105 9106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:35.056336 9106 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:35.056499 9106 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5c480826a345432c88a500716f5d164d: Loaded TSK: 0
I20260928 23:17:35.056707 9106 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:17:35.123090 9098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:35.123277 9098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:35.123301 9098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:35.124483 9098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:35.124522 9098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:35.124547 9098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:35.125941 9098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45371
--tserver_master_addrs=127.0.98.62:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:35.126148 9098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:35.126358 9098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:35.130929 9122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:35.130952 9125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:35.130985 9123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:35.131161 9098 server_base.cc:1034] running on GCE node
I20260928 23:17:35.131390 9098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:35.131690 9098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:35.132879 9098 hybrid_clock.cc:648] HybridClock initialized: now 1790637455132854 us; error 35 us; skew 500 ppm
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637455, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:35.135907 9098 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 23:17:35.136451 9098 webserver.cc:460] Webserver started at http://127.0.98.1:45371/ using document root <none> and password file <none>
I20260928 23:17:35.136720 9098 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:35.136796 9098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:35.138474 9098 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:17:35.139061 9132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.139225 9098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:35.139321 9098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "2616aeb287c54c269fca354925654a11"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "3aa472465266604f634578816a88a840"
server_key_iv: "c7c13585e790fb3631ed089ec6abec7f"
server_key_version: "encryptionkey@0"
I20260928 23:17:35.139665 9098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:35.147539 9098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:35.149528 9098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:35.149634 9098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:35.149832 9098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:35.150100 9098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:35.150132 9098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.150161 9098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:35.150184 9098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.155361 9098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32999
I20260928 23:17:35.155457 9245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32999 every 8 connection(s)
I20260928 23:17:35.155743 9098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:35.158495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 9098
I20260928 23:17:35.158784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38815
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42319
--webserver_port=37799
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:35.164443 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:35.156418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:34677)
0928 23:17:35.156595 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:35.156641 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:35.157248 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:35.157838 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:35.157842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:35.158359 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:35.158539 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.158544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.159748 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.159749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.160867 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.160870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.161686 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.161687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.161792 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.162234 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:35.162242 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:35.164234 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
W20260928 23:17:35.164700 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:35.156520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34677 (local address 127.0.98.62:42319)
0928 23:17:35.157443 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:35.157451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:35.157485 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:17:35.157602 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:35.157608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:35.157661 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:35.157813 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:35.158667 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.159656 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.160965 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.161589 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.161710 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.164368 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34677: BlockingRecv error: recv got EOF from 127.0.98.1:34677 (error 108)
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":72,"threads_started":1}
W20260928 23:17:35.164719 9246 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42319 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42319: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:35.168655 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:35.164928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:41405)
0928 23:17:35.164968 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:35.164976 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:35.165006 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:35.165135 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:35.165136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:35.165244 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:35.165293 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.165294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.165890 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.165891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.166383 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.166384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.166698 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.166699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.166758 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.167098 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:35.167103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:35.168594 (+ 1491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 23:17:35.168730 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:35.164962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41405 (local address 127.0.98.62:42319)
0928 23:17:35.164999 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:35.165001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:35.165010 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:35.165051 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:35.165052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:35.165060 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:35.165120 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:35.165355 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.165826 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.166450 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.166645 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.166694 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.168678 (+ 1984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41405: BlockingRecv error: recv got EOF from 127.0.98.1:41405 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:35.172583 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:35.169045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:36603)
0928 23:17:35.169130 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:35.169139 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:35.169180 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:35.169281 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:35.169282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:35.169395 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:35.169418 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.169419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.169970 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.169971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.170447 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.170448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.170734 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.170735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.170781 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.171117 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:35.171122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:35.172526 (+ 1404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 23:17:35.172650 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:35.169099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36603 (local address 127.0.98.62:42319)
0928 23:17:35.169131 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:35.169133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:35.169168 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:17:35.169231 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:35.169232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:35.169240 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:35.169278 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:35.169477 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.169907 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.170509 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.170682 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.170732 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.172602 (+ 1870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36603: BlockingRecv error: recv got EOF from 127.0.98.1:36603 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260928 23:17:35.172888 9246 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:17:35.245436 9250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:35.245626 9250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:35.245647 9250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:35.246723 9250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:35.246755 9250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:35.246779 9250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:35.248132 9250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37799
--tserver_master_addrs=127.0.98.62:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:35.248311 9250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:35.248478 9250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:35.253002 9256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:35.252997 9255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:35.253050 9250 server_base.cc:1034] running on GCE node
W20260928 23:17:35.252993 9258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:35.253345 9250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:35.253535 9250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:35.254668 9250 hybrid_clock.cc:648] HybridClock initialized: now 1790637455254657 us; error 25 us; skew 500 ppm
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637455, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:35.256945 9250 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 23:17:35.257360 9250 webserver.cc:460] Webserver started at http://127.0.98.2:37799/ using document root <none> and password file <none>
I20260928 23:17:35.257541 9250 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:35.257594 9250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:35.258781 9250 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:35.259249 9265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.259387 9250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.259469 9250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "c5a294a373c34f6d933fb39a63b15e24"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "52162d0022871d6206782f54ad3f4fd5"
server_key_iv: "77a8eaa116bcbacd048da7530b5b9aa4"
server_key_version: "encryptionkey@0"
I20260928 23:17:35.259804 9250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:35.270648 9250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:35.272260 9250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:35.272368 9250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:35.272527 9250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:35.272747 9250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:35.272773 9250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.272794 9250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:35.272809 9250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.278254 9250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38815
I20260928 23:17:35.278293 9378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38815 every 8 connection(s)
I20260928 23:17:35.278602 9250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:35.284512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 9250
I20260928 23:17:35.284806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45267
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42319
--webserver_port=42807
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:35.285360 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:35.279236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45327 (local address 127.0.98.62:42319)
0928 23:17:35.279307 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:35.279309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:35.279498 (+ 189us) server_negotiation.cc:407] Connection header received
0928 23:17:35.280010 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:35.280011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:35.280025 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:35.280075 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:35.280917 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.281398 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.282301 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.282485 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.282538 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.285313 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45327: BlockingRecv error: recv got EOF from 127.0.98.2:45327 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:17:35.285470 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:35.279227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:45327)
0928 23:17:35.279386 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:35.279420 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:35.279928 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:35.280131 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:35.280134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:35.280658 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:35.280850 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.280855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.281459 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.281459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.282231 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.282232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.282545 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.282546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.282639 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.283046 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:35.283053 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:35.285230 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
W20260928 23:17:35.285663 9379 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42319 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42319: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:35.289785 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:35.285855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:55373)
0928 23:17:35.285950 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:35.285961 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:35.286014 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:35.286136 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:35.286137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:35.286239 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:35.286278 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.286279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.286877 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.286878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.287361 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.287363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.287734 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.287735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.287796 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.288176 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:35.288180 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:35.289727 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 23:17:35.289866 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:35.285908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55373 (local address 127.0.98.62:42319)
0928 23:17:35.285956 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:35.285959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:35.286001 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:17:35.286065 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:35.286065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:35.286076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:35.286131 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:35.286354 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.286813 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.287428 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.287671 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.287719 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.289815 (+ 2096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55373: BlockingRecv error: recv got EOF from 127.0.98.2:55373 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:35.293836 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:35.289988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:45021)
0928 23:17:35.290109 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:35.290117 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:35.290154 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:35.290267 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:35.290268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:35.290377 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:35.290400 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.290401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.290961 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.290962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.291439 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.291441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.291799 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.291800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.291848 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.292220 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:35.292225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:35.293767 (+ 1542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
W20260928 23:17:35.293895 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:35.290026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45021 (local address 127.0.98.62:42319)
0928 23:17:35.290054 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:35.290056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:35.290157 (+ 101us) server_negotiation.cc:407] Connection header received
0928 23:17:35.290205 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:35.290206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:35.290214 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:35.290264 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:35.290457 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.290900 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.291504 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.291738 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.291792 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.293845 (+ 2053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45021: BlockingRecv error: recv got EOF from 127.0.98.2:45021 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260928 23:17:35.293977 9379 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:17:35.363080 9382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:35.363270 9382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:35.363291 9382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:35.364444 9382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:35.364485 9382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:35.364508 9382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:35.365918 9382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42807
--tserver_master_addrs=127.0.98.62:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:35.366134 9382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:35.366334 9382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:17:35.370930 9382 server_base.cc:1034] running on GCE node
W20260928 23:17:35.370913 9388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:35.370908 9391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:35.370913 9389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:35.371300 9382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:35.371515 9382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:35.372668 9382 hybrid_clock.cc:648] HybridClock initialized: now 1790637455372655 us; error 27 us; skew 500 ppm
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637455, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:35.374927 9382 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 23:17:35.375272 9382 webserver.cc:460] Webserver started at http://127.0.98.3:42807/ using document root <none> and password file <none>
I20260928 23:17:35.375450 9382 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:35.375499 9382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:35.376746 9382 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:35.377240 9398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.377372 9382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.377449 9382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "2af72b52bfaf4e2196759774c66f715f"
format_stamp: "Formatted at 2026-09-28 23:17:34 on dist-test-slave-djkr"
server_key: "20ab26228e78f0e04c01507a3419449b"
server_key_iv: "542852c2673b18138389c5c63f450417"
server_key_version: "encryptionkey@0"
I20260928 23:17:35.377739 9382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:35.406366 9382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:35.408155 9382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:35.408293 9382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:35.408499 9382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:35.408769 9382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:35.408809 9382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.408839 9382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:35.408860 9382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:35.414065 9382 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45267
I20260928 23:17:35.414121 9511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45267 every 8 connection(s)
I20260928 23:17:35.414423 9382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:35.419600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 9382
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:35.421301 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:35.415092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:56723)
0928 23:17:35.415244 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:35.415278 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:35.415810 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:35.415999 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:35.416003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:35.416525 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:35.416708 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.416713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.417365 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.417366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.418137 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.418139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.418453 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.418454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.418546 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.418914 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:35.418921 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:35.421162 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
W20260928 23:17:35.421312 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:35.415088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56723 (local address 127.0.98.62:42319)
0928 23:17:35.415157 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:35.415159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:35.415371 (+ 212us) server_negotiation.cc:407] Connection header received
0928 23:17:35.415884 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:35.415885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:35.415895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:35.415944 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:35.416817 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.417302 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.418212 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.418392 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.418443 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.421253 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56723: BlockingRecv error: recv got EOF from 127.0.98.3:56723 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 23:17:35.421512 9512 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42319 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42319: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:35.425311 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:35.421697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:51179)
0928 23:17:35.421809 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:35.421818 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:35.421864 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:35.421961 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:35.421962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:35.422090 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:35.422131 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.422132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.422685 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.422686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.423158 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.423159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.423450 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.423450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.423507 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.423863 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:35.423867 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:35.425253 (+ 1386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
W20260928 23:17:35.425375 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:35.421733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51179 (local address 127.0.98.62:42319)
0928 23:17:35.421765 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:35.421767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:35.421861 (+ 94us) server_negotiation.cc:407] Connection header received
0928 23:17:35.421913 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:35.421913 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:35.421920 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:35.421957 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:35.422196 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.422607 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.423226 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.423398 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.423447 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.425326 (+ 1879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51179: BlockingRecv error: recv got EOF from 127.0.98.3:51179 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:17:35.426990 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:35.420091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.0.98.62:42319)
0928 23:17:35.420201 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:35.420203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:35.420211 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:35.420252 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:35.420253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:35.420264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:35.420328 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:35.421046 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.421639 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.422532 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.422758 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.422809 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.423986 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:35.423991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:35.423992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:35.424002 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:35.425751 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:35.426087 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:35.426090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:35.426090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:35.426190 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:35.426350 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:35.426351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:35.426351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:35.426613 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:35.426738 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:35.426799 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 23:17:35.426903 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:35 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:35.429669 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:35.425531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:40725)
0928 23:17:35.425828 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:35.425837 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:35.425876 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:35.425976 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:35.425977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:35.426095 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:35.426122 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.426124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.426863 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.426864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.427417 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:35.427419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.427837 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.427837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:35.427880 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.428179 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:35.428183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:35.429615 (+ 1432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283}
W20260928 23:17:35.429728 9249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:35.425572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40725 (local address 127.0.98.62:42319)
0928 23:17:35.425600 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:35.425601 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:35.425866 (+ 265us) server_negotiation.cc:407] Connection header received
0928 23:17:35.425924 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:35.425924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:35.425931 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:35.425970 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:35.426178 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.426795 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.427497 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:35.427777 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:35.427831 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:35.429681 (+ 1850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40725: BlockingRecv error: recv got EOF from 127.0.98.3:40725 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260928 23:17:35.429787 9512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 23:17:36 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:36 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:36.177212 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:36.173189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:51039)
0928 23:17:36.173340 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:36.173351 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:36.173411 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:36.173526 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:36.173526 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:36.173666 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:36.173713 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:36.173714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.174304 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.174304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:36.174841 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:36.174843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.175121 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.175122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:36.175155 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:36.175587 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:36.175591 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:36.177147 (+ 1556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
W20260928 23:17:36.177296 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:36.173242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51039 (local address 127.0.98.62:42319)
0928 23:17:36.173345 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:36.173347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:36.173398 (+ 51us) server_negotiation.cc:407] Connection header received
0928 23:17:36.173451 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:36.173451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:36.173460 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:36.173514 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:36.173765 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.174251 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.174897 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.175077 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.175118 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:36.177231 (+ 2113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51039: BlockingRecv error: recv got EOF from 127.0.98.1:51039 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
Sep 28 23:17:36 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:36 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:36.298575 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:36.294350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:46663)
0928 23:17:36.294478 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:36.294489 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:36.294549 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:36.294744 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:36.294745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:36.294905 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:36.294951 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:36.294952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.295625 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.295626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:36.296189 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:36.296191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.296543 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.296544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:36.296580 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:36.296971 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:36.296975 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:36.298503 (+ 1528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
W20260928 23:17:36.298656 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:36.294415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46663 (local address 127.0.98.62:42319)
0928 23:17:36.294480 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:36.294482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:36.294639 (+ 157us) server_negotiation.cc:407] Connection header received
0928 23:17:36.294665 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:36.294666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:36.294678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:36.294734 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:36.295013 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.295517 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.296267 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.296487 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.296541 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:36.298600 (+ 2059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46663: BlockingRecv error: recv got EOF from 127.0.98.2:46663 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 23:17:36 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:36 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:36.434080 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:36.430076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:35039)
0928 23:17:36.430178 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:36.430188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:36.430234 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:36.430386 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:36.430386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:36.430530 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:36.430575 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:36.430577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.431135 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.431135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:36.431728 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:36.431730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.432008 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.432008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:36.432045 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:36.432436 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:36.432440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:36.434019 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
W20260928 23:17:36.434219 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:36.430132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35039 (local address 127.0.98.62:42319)
0928 23:17:36.430208 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:36.430210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:36.430220 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:36.430318 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:36.430318 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:36.430329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:36.430373 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:36.430619 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.431087 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.431789 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:36.431962 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:36.432003 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:36.434152 (+ 2149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35039: BlockingRecv error: recv got EOF from 127.0.98.3:35039 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 23:17:37 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:37 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:37.182031 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:37.177651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:46173)
0928 23:17:37.177795 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:37.177804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:37.177860 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:37.177986 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:37.177987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:37.178143 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:37.178197 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:37.178198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.178814 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.178815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:37.179351 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:37.179352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.179690 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.179690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:37.179749 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:37.180151 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:37.180156 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:37.181972 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
W20260928 23:17:37.182127 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:37.177698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46173 (local address 127.0.98.62:42319)
0928 23:17:37.177800 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:37.177801 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:37.177854 (+ 53us) server_negotiation.cc:407] Connection header received
0928 23:17:37.177901 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:37.177901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:37.177911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:37.177969 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:37.178253 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.178754 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.179408 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.179632 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.179682 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:37.182057 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46173: BlockingRecv error: recv got EOF from 127.0.98.1:46173 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
Sep 28 23:17:37 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:37 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:37.303459 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:37.299080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:60713)
0928 23:17:37.299195 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:37.299206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:37.299278 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:37.299421 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:37.299422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:37.299629 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:37.299691 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:37.299693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.300335 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.300336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:37.300874 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:37.300876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.301263 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.301264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:37.301312 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:37.301778 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:37.301782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:37.303390 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
W20260928 23:17:37.303535 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:37.299132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60713 (local address 127.0.98.62:42319)
0928 23:17:37.299177 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:37.299179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:37.299265 (+ 86us) server_negotiation.cc:407] Connection header received
0928 23:17:37.299338 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:37.299339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:37.299352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:37.299406 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:37.299763 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.300246 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.300969 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.301158 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.301253 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:37.303477 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60713: BlockingRecv error: recv got EOF from 127.0.98.2:60713 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 23:17:37 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:37 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:37.438673 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:37.434500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:55395)
0928 23:17:37.434641 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:37.434654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:37.434710 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:37.434820 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:37.434821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:37.434975 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:37.435021 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:37.435022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.435622 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.435623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:37.436150 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:37.436151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.436446 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.436447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:37.436482 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:37.436875 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:37.436879 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:37.438615 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
W20260928 23:17:37.438752 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:37.434623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55395 (local address 127.0.98.62:42319)
0928 23:17:37.434663 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:37.434665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:37.434696 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:17:37.434759 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:37.434759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:37.434770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:37.434809 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:37.435070 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.435575 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.436217 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:37.436400 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:37.436445 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:37.438693 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55395: BlockingRecv error: recv got EOF from 127.0.98.3:55395 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 23:17:38 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:38 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:38.186779 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:38.182448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:58267)
0928 23:17:38.182582 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:38.182592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:38.182638 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:38.182761 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:38.182761 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:38.182911 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:38.182957 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:38.182959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.183609 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.183611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:38.184147 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:38.184149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.184440 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.184441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:38.184477 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:38.184888 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:38.184892 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:38.186712 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
W20260928 23:17:38.186856 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:38.182519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58267 (local address 127.0.98.62:42319)
0928 23:17:38.182643 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:38.182645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:38.182654 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:38.182675 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:38.182676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:38.182685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:38.182751 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:38.183010 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.183502 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.184208 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.184390 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.184433 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:38.186791 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58267: BlockingRecv error: recv got EOF from 127.0.98.1:58267 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
Sep 28 23:17:38 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:38 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:38.308252 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:38.303908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:39235)
0928 23:17:38.304043 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:38.304056 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:38.304127 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:38.304266 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:38.304267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:38.304447 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:38.304494 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:38.304496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.305122 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.305123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:38.305657 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:38.305658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.305945 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.305945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:38.305981 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:38.306364 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:38.306368 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:38.308165 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
W20260928 23:17:38.308306 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:38.303966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39235 (local address 127.0.98.62:42319)
0928 23:17:38.304018 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:38.304021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:38.304110 (+ 89us) server_negotiation.cc:407] Connection header received
0928 23:17:38.304183 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:38.304184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:38.304194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:38.304263 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:38.304544 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.305045 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.305718 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.305896 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.305941 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:38.308251 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39235: BlockingRecv error: recv got EOF from 127.0.98.2:39235 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 23:17:38 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:38 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:38.443372 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:38.439132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:41653)
0928 23:17:38.439288 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:38.439301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:38.439367 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:38.439489 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:38.439490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:38.439665 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:38.439710 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:38.439713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.440307 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.440308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:38.440832 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:38.440834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.441144 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.441145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:38.441179 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:38.441574 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:38.441578 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:38.443310 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
W20260928 23:17:38.443447 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:38.439226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41653 (local address 127.0.98.62:42319)
0928 23:17:38.439310 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:38.439312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:38.439359 (+ 47us) server_negotiation.cc:407] Connection header received
0928 23:17:38.439423 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:38.439424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:38.439435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:38.439473 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:38.439773 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.440253 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.440909 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:38.441093 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:38.441144 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:38.443386 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41653: BlockingRecv error: recv got EOF from 127.0.98.3:41653 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 23:17:39 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:39 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:39.191606 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:39.187229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:34043)
0928 23:17:39.187384 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:39.187398 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:39.187460 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:39.187621 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:39.187621 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:39.187790 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:39.187840 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:39.187841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.188443 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.188444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:39.189000 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:39.189002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.189371 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.189373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:39.189430 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:39.189910 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:39.189915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:39.191503 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
W20260928 23:17:39.191702 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:39.187317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34043 (local address 127.0.98.62:42319)
0928 23:17:39.187463 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:39.187465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:39.187477 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:17:39.187501 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:39.187502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:39.187512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:39.187614 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:39.187893 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.188386 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.189072 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.189307 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.189365 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:39.191621 (+ 2256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34043: BlockingRecv error: recv got EOF from 127.0.98.1:34043 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
Sep 28 23:17:39 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:39 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:39.313009 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:39.308739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:54843)
0928 23:17:39.308874 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:39.308884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:39.308943 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:39.309137 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:39.309138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:39.309319 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:39.309369 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:39.309370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.309973 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.309973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:39.310502 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:39.310504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.310798 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.310799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:39.310843 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:39.311295 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:39.311300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:39.312949 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
W20260928 23:17:39.313099 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:39.308792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54843 (local address 127.0.98.62:42319)
0928 23:17:39.308838 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:39.308839 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:39.308935 (+ 96us) server_negotiation.cc:407] Connection header received
0928 23:17:39.309037 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:39.309037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:39.309048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:39.309096 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:39.309427 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.309913 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.310556 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.310740 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.310784 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:39.313037 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54843: BlockingRecv error: recv got EOF from 127.0.98.2:54843 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 23:17:39 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:39 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:39.455294 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:39.443779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:60595)
0928 23:17:39.443892 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:39.443902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:39.443960 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:39.444078 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:39.444078 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:39.444232 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:39.444279 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:39.444280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.444897 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.444898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:39.445500 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:39.445503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.445828 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.445829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:39.445866 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:39.453482 (+ 7616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:39.453488 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:39.455218 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
W20260928 23:17:39.455404 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:39.443834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60595 (local address 127.0.98.62:42319)
0928 23:17:39.443907 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:39.443909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:39.443957 (+ 48us) server_negotiation.cc:407] Connection header received
0928 23:17:39.444015 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:39.444016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:39.444027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:39.444066 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:39.444331 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.444835 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.445575 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:39.445774 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:39.445825 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:39.455348 (+ 9523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60595: BlockingRecv error: recv got EOF from 127.0.98.3:60595 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 23:17:40 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:40 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:40.196429 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:40.192111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:37681)
0928 23:17:40.192256 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:40.192266 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:40.192340 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:40.192512 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:40.192513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:40.192718 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:40.192778 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:40.192781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.193419 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.193420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:40.193935 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:40.193936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.194236 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.194237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:40.194273 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:40.194678 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:40.194682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:40.196371 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
W20260928 23:17:40.196525 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:40.192159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37681 (local address 127.0.98.62:42319)
0928 23:17:40.192272 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:40.192274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:40.192334 (+ 60us) server_negotiation.cc:407] Connection header received
0928 23:17:40.192418 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:40.192419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:40.192431 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:40.192491 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:40.192842 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.193362 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.193986 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.194187 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.194230 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:40.196459 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37681: BlockingRecv error: recv got EOF from 127.0.98.1:37681 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
Sep 28 23:17:40 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:40 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:40.317610 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:40.313453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:33085)
0928 23:17:40.313581 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:40.313591 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:40.313651 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:40.313768 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:40.313769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:40.313930 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:40.313984 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:40.313985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.314632 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.314633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:40.315155 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:40.315157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.315481 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.315482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:40.315535 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:40.315951 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:40.315955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:40.317532 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
W20260928 23:17:40.317687 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:40.313598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33085 (local address 127.0.98.62:42319)
0928 23:17:40.313662 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:40.313663 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:40.313674 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:17:40.313702 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:40.313702 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:40.313714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:40.313758 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:40.314058 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.314540 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.315234 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.315423 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.315476 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:40.317633 (+ 2157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33085: BlockingRecv error: recv got EOF from 127.0.98.2:33085 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 23:17:40 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:40 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:40.460089 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:40.455774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:42967)
0928 23:17:40.455881 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:40.455892 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:40.455953 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:40.456069 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:40.456070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:40.456263 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:40.456311 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:40.456313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.456914 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.456914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:40.457482 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:40.457484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.457778 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.457779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:40.457838 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:40.458332 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:40.458337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:40.460033 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
W20260928 23:17:40.460163 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:40.455838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42967 (local address 127.0.98.62:42319)
0928 23:17:40.455902 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:40.455904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:40.455944 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:17:40.456006 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:40.456007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:40.456018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:40.456055 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:40.456362 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.456846 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.457540 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:40.457708 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:40.457750 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:40.460104 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42967: BlockingRecv error: recv got EOF from 127.0.98.3:42967 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 23:17:41 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:41 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:41.201104 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:41.196861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:44147)
0928 23:17:41.196984 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:41.196996 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:41.197042 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:41.197180 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:41.197180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:41.197336 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:41.197382 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:41.197383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.198101 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.198102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:41.198646 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:41.198647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.198958 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.198958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:41.198995 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:41.199409 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:41.199413 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:41.201036 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
W20260928 23:17:41.201231 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:41.196908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44147 (local address 127.0.98.62:42319)
0928 23:17:41.197031 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:41.197033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:41.197042 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:41.197093 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:41.197093 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:41.197104 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:41.197165 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:41.197508 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.198034 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.198713 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.198909 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.198956 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:41.201151 (+ 2195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44147: BlockingRecv error: recv got EOF from 127.0.98.1:44147 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
Sep 28 23:17:41 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:41 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:41.322292 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:41.318049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:39525)
0928 23:17:41.318153 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:41.318170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:41.318226 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:41.318327 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:41.318327 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:41.318479 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:41.318523 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:41.318524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.319096 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.319096 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:41.319680 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:41.319683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.320098 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.320099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:41.320163 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:41.320593 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:41.320598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:41.322198 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":53,"threads_started":1}
W20260928 23:17:41.322348 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:41.318105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39525 (local address 127.0.98.62:42319)
0928 23:17:41.318175 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:41.318176 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:41.318213 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:17:41.318276 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:41.318277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:41.318287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:41.318321 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:41.318570 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.319045 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.319748 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.320035 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.320097 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:41.322297 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39525: BlockingRecv error: recv got EOF from 127.0.98.2:39525 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 23:17:41 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:41 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:41.464800 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:41.460474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:35313)
0928 23:17:41.460590 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:41.460601 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:41.460666 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:41.460803 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:41.460804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:41.460970 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:41.461013 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:41.461016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.461646 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.461647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:41.462212 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:41.462214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.462540 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.462541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:41.462594 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:41.463021 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:41.463026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:41.464711 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
W20260928 23:17:41.464864 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:41.460538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35313 (local address 127.0.98.62:42319)
0928 23:17:41.460594 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:41.460596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:41.460653 (+ 57us) server_negotiation.cc:407] Connection header received
0928 23:17:41.460719 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:41.460720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:41.460733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:41.460795 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:41.461077 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.461573 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.462282 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:41.462485 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:41.462537 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:41.464807 (+ 2270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35313: BlockingRecv error: recv got EOF from 127.0.98.3:35313 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 23:17:42 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:42 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:42.205896 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:42.201534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:35487)
0928 23:17:42.201646 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:42.201656 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:42.201713 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:42.201843 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:42.201844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:42.201998 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:42.202044 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:42.202046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.202713 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.202714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:42.203245 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:42.203246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.203615 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.203616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:42.203672 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:42.204092 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:42.204096 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:42.205837 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
W20260928 23:17:42.206003 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:42.201626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35487 (local address 127.0.98.62:42319)
0928 23:17:42.201732 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:42.201734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:42.201743 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:42.201766 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:42.201766 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:42.201776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:42.201830 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:42.202107 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.202636 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.203325 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.203528 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.203610 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:42.205928 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35487: BlockingRecv error: recv got EOF from 127.0.98.1:35487 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
Sep 28 23:17:42 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:42 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:42.327154 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:42.322746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:43115)
0928 23:17:42.322865 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:42.322877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:42.322942 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:42.323067 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:42.323069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:42.323236 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:42.323281 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:42.323283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.323935 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.323936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:42.324500 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:42.324503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.324818 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.324819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:42.324874 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:42.325299 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:42.325304 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:42.327067 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
W20260928 23:17:42.327222 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:42.322808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43115 (local address 127.0.98.62:42319)
0928 23:17:42.322871 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:42.322873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:42.322925 (+ 52us) server_negotiation.cc:407] Connection header received
0928 23:17:42.322994 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:42.322995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:42.323006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:42.323055 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:42.323347 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.323868 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.324579 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.324761 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.324813 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:42.327162 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43115: BlockingRecv error: recv got EOF from 127.0.98.2:43115 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 23:17:42 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:42 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:42.469425 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:42.465226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:51263)
0928 23:17:42.465348 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:42.465358 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:42.465435 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:42.465566 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:42.465567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:42.465727 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:42.465788 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:42.465790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.466411 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.466412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:42.466932 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:42.466933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.467239 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.467239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:42.467292 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:42.467757 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:42.467766 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:42.469368 (+ 1602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
W20260928 23:17:42.469503 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:42.465347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51263 (local address 127.0.98.62:42319)
0928 23:17:42.465402 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:42.465403 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:42.465412 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:42.465497 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:42.465497 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:42.465510 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:42.465552 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:42.465854 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.466342 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.466997 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:42.467181 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:42.467240 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:42.469448 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51263: BlockingRecv error: recv got EOF from 127.0.98.3:51263 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 23:17:43 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:43 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:43.210412 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:43.206282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:50753)
0928 23:17:43.206417 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:43.206428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:43.206485 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:43.206638 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:43.206638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:43.206792 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:43.206838 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:43.206840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.207491 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.207492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:43.208023 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:43.208024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.208324 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.208324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:43.208359 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:43.208782 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:43.208786 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:43.210338 (+ 1552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":53,"threads_started":1}
W20260928 23:17:43.210492 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:43.206383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50753 (local address 127.0.98.62:42319)
0928 23:17:43.206522 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:43.206524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:43.206532 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:43.206553 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:43.206554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:43.206564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:43.206627 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:43.206919 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.207435 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.208077 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.208274 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.208322 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:43.210432 (+ 2110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50753: BlockingRecv error: recv got EOF from 127.0.98.1:50753 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
Sep 28 23:17:43 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:43 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:43.331909 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:43.327642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:36983)
0928 23:17:43.327795 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:43.327805 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:43.327872 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:43.327998 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:43.327999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:43.328167 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:43.328210 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:43.328211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.328969 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.328970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:43.329493 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:43.329494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.329820 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.329821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:43.329876 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:43.330261 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:43.330265 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:43.331834 (+ 1569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
W20260928 23:17:43.331974 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:43.327716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36983 (local address 127.0.98.62:42319)
0928 23:17:43.327788 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:43.327790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:43.327864 (+ 74us) server_negotiation.cc:407] Connection header received
0928 23:17:43.327924 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:43.327925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:43.327937 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:43.327985 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:43.328280 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.328901 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.329566 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.329757 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.329809 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:43.331922 (+ 2113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36983: BlockingRecv error: recv got EOF from 127.0.98.2:36983 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 23:17:43 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:43 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:43.474465 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:43.469971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:50193)
0928 23:17:43.470110 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:43.470119 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:43.470175 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:43.470321 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:43.470322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:43.470477 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:43.470530 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:43.470531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.471358 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.471359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:43.471902 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:43.471904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.472207 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.472208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:43.472275 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:43.472721 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:43.472725 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:43.474405 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
W20260928 23:17:43.474555 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:43.470056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50193 (local address 127.0.98.62:42319)
0928 23:17:43.470114 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:43.470116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:43.470170 (+ 54us) server_negotiation.cc:407] Connection header received
0928 23:17:43.470220 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:43.470221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:43.470234 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:43.470305 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:43.470578 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.471251 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.471965 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:43.472142 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:43.472184 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:43.474489 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50193: BlockingRecv error: recv got EOF from 127.0.98.3:50193 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 23:17:44 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:44 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:44.215132 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:44.210860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:53121)
0928 23:17:44.210967 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:44.210978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:44.211040 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:44.211181 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:44.211182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:44.211334 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:44.211384 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:44.211385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.212032 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.212033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:44.212560 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:44.212562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.212881 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.212882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:44.212926 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:44.213337 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:44.213341 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:44.215056 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
W20260928 23:17:44.215215 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:44.210928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53121 (local address 127.0.98.62:42319)
0928 23:17:44.211079 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:44.211080 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:44.211088 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:44.211108 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:44.211108 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:44.211118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:44.211170 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:44.211436 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.211947 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.212632 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.212825 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.212873 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:44.215144 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53121: BlockingRecv error: recv got EOF from 127.0.98.1:53121 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
Sep 28 23:17:44 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:44 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:44.337102 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:44.332414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:37079)
0928 23:17:44.332575 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:44.332589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:44.332666 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:44.332798 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:44.332800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:44.332969 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:44.333016 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:44.333017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.333640 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.333641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:44.334194 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:44.334196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.334526 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.334527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:44.334582 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:44.334991 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:44.334996 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:44.337044 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
W20260928 23:17:44.337168 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:44.332498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37079 (local address 127.0.98.62:42319)
0928 23:17:44.332579 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:44.332582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:44.332667 (+ 85us) server_negotiation.cc:407] Connection header received
0928 23:17:44.332720 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:44.332721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:44.332733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:44.332780 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:44.333084 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.333568 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.334273 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.334466 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.334520 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:44.337110 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37079: BlockingRecv error: recv got EOF from 127.0.98.2:37079 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 23:17:44 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:44 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:44.482908 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:44.474877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:33823)
0928 23:17:44.475019 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:44.475029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:44.475090 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:44.475211 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:44.475212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:44.475394 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:44.475450 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:44.475452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.476096 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.476098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:44.476628 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:44.476630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.476936 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.476937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:44.476990 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:44.481181 (+ 4191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:44.481186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:44.482851 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
W20260928 23:17:44.482995 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:44.474943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33823 (local address 127.0.98.62:42319)
0928 23:17:44.474994 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:44.474997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:44.475079 (+ 82us) server_negotiation.cc:407] Connection header received
0928 23:17:44.475145 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:44.475146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:44.475159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:44.475201 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:44.475515 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.476030 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.476698 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:44.476881 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:44.476932 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:44.482929 (+ 5997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33823: BlockingRecv error: recv got EOF from 127.0.98.3:33823 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 23:17:45 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:45 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:45.219676 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:45.215603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:36901)
0928 23:17:45.215721 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:45.215732 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:45.215797 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:45.215911 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:45.215911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:45.216064 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:45.216111 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:45.216112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.216695 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.216696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:45.217228 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:45.217229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.217506 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.217506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:45.217543 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:45.217969 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:45.217973 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:45.219600 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
W20260928 23:17:45.219765 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:45.215657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36901 (local address 127.0.98.62:42319)
0928 23:17:45.215793 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:45.215795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:45.215803 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:45.215840 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:45.215840 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:45.215850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:45.215901 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:45.216159 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.216645 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.217281 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.217460 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.217503 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:45.219702 (+ 2199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36901: BlockingRecv error: recv got EOF from 127.0.98.1:36901 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
Sep 28 23:17:45 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:45 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:45.342001 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:45.337484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:51329)
0928 23:17:45.337638 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:45.337651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:45.337723 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:45.337859 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:45.337860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:45.338053 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:45.338101 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:45.338103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.338740 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.338740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:45.339277 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:45.339278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.339624 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.339625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:45.339681 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:45.340120 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:45.340125 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:45.341924 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
W20260928 23:17:45.342057 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:45.337538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51329 (local address 127.0.98.62:42319)
0928 23:17:45.337594 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:45.337596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:45.337708 (+ 112us) server_negotiation.cc:407] Connection header received
0928 23:17:45.337777 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:45.337778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:45.337791 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:45.337842 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:45.338162 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.338678 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.339334 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.339507 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.339567 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:45.342005 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51329: BlockingRecv error: recv got EOF from 127.0.98.2:51329 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 23:17:45 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:45 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:45.487643 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:45.483349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:54963)
0928 23:17:45.483483 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:45.483494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:45.483569 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:45.483700 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:45.483701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:45.483856 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:45.483903 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:45.483904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.484496 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.484497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:45.485032 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:45.485033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.485318 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.485319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:45.485356 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:45.485800 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:45.485804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:45.487509 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":55,"threads_started":1}
W20260928 23:17:45.487681 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:45.483464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54963 (local address 127.0.98.62:42319)
0928 23:17:45.483520 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:45.483522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:45.483533 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:17:45.483632 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:45.483632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:45.483643 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:45.483687 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:45.483951 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.484436 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.485098 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:45.485273 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:45.485314 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:45.487619 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54963: BlockingRecv error: recv got EOF from 127.0.98.3:54963 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 23:17:46 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:46 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:46.224447 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:46.220158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:49457)
0928 23:17:46.220307 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:46.220319 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:46.220372 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:46.220532 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:46.220533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:46.220697 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:46.220742 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:46.220744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.221370 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.221371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:46.221890 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:46.221891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.222213 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.222214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:46.222273 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:46.222685 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:46.222690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:46.224381 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
W20260928 23:17:46.224543 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:46.220225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49457 (local address 127.0.98.62:42319)
0928 23:17:46.220347 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:46.220349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:46.220357 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:46.220439 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:46.220440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:46.220453 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:46.220519 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:46.220816 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.221302 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.221967 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.222156 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.222208 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:46.224475 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49457: BlockingRecv error: recv got EOF from 127.0.98.1:49457 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
Sep 28 23:17:46 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:46 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:46.346627 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:46.342478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:44571)
0928 23:17:46.342596 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:46.342606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:46.342660 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:46.342776 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:46.342777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:46.342923 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:46.343003 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:46.343005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.343641 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.343641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:46.344165 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:46.344167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.344515 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.344516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:46.344564 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:46.344951 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:46.344955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:46.346568 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
W20260928 23:17:46.346722 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:46.342528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44571 (local address 127.0.98.62:42319)
0928 23:17:46.342580 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:46.342582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:46.342654 (+ 72us) server_negotiation.cc:407] Connection header received
0928 23:17:46.342703 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:46.342704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:46.342715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:46.342764 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:46.343058 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.343585 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.344233 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.344448 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.344500 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:46.346662 (+ 2162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44571: BlockingRecv error: recv got EOF from 127.0.98.2:44571 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 23:17:46 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:46 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:46.492187 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:46.488060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:49621)
0928 23:17:46.488222 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:46.488233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:46.488296 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:46.488451 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:46.488452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:46.488619 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:46.488665 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:46.488667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.489271 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.489272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:46.489802 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:46.489804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.490070 (+ 266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.490070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:46.490104 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:46.490503 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:46.490507 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:46.492129 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
W20260928 23:17:46.492283 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:46.488132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49621 (local address 127.0.98.62:42319)
0928 23:17:46.488200 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:46.488202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:46.488280 (+ 78us) server_negotiation.cc:407] Connection header received
0928 23:17:46.488350 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:46.488350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:46.488363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:46.488408 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:46.488724 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.489205 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.489859 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:46.490026 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:46.490067 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:46.492221 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49621: BlockingRecv error: recv got EOF from 127.0.98.3:49621 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 23:17:47 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:47 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:47.229090 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:47.224914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:38067)
0928 23:17:47.225035 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:47.225045 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:47.225104 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:47.225230 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:47.225231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:47.225395 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:47.225443 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:47.225445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.226123 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.226124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:47.226640 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:47.226642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.226949 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.226950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:47.226984 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:47.227407 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:47.227412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:47.229034 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
W20260928 23:17:47.229198 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:47.225002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38067 (local address 127.0.98.62:42319)
0928 23:17:47.225117 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:47.225119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:47.225127 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:47.225150 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:47.225151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:47.225161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:47.225221 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:47.225532 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.226060 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.226702 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.226894 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.226945 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:47.229130 (+ 2185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38067: BlockingRecv error: recv got EOF from 127.0.98.1:38067 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
Sep 28 23:17:47 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:47 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:47.351461 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:47.347077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:52367)
0928 23:17:47.347201 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:47.347211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:47.347267 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:47.347417 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:47.347418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:47.347615 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:47.347664 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:47.347666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.348455 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.348456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:47.348978 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:47.348980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.349305 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.349306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:47.349343 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:47.349723 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:47.349728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:47.351392 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
W20260928 23:17:47.351529 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:47.347170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52367 (local address 127.0.98.62:42319)
0928 23:17:47.347221 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:47.347222 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:47.347263 (+ 41us) server_negotiation.cc:407] Connection header received
0928 23:17:47.347310 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:47.347310 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:47.347321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:47.347382 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:47.347744 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.348391 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.349043 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.349251 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.349302 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:47.351470 (+ 2168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52367: BlockingRecv error: recv got EOF from 127.0.98.2:52367 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 23:17:47 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:47 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:47.496913 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:47.492603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:35191)
0928 23:17:47.492739 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:47.492750 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:47.492813 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:47.492970 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:47.492971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:47.493165 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:47.493210 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:47.493213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.493841 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.493842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:47.494367 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:47.494369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.494689 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.494690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:47.494742 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:47.495155 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:47.495159 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:47.496834 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
W20260928 23:17:47.496996 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:47.492709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35191 (local address 127.0.98.62:42319)
0928 23:17:47.492761 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:47.492762 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:47.492796 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:17:47.492891 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:47.492892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:47.492906 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:47.492951 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:47.493287 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.493775 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.494442 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:47.494630 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:47.494684 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:47.496932 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35191: BlockingRecv error: recv got EOF from 127.0.98.3:35191 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 23:17:48 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:48 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:48.233857 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:48.229543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:45407)
0928 23:17:48.229672 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:48.229683 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:48.229746 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:48.229860 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:48.229861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:48.230014 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:48.230060 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:48.230062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.230661 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.230662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:48.231189 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:48.231191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.231607 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.231608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:48.231659 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:48.232074 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:48.232078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:48.233769 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
W20260928 23:17:48.233927 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:48.229629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45407 (local address 127.0.98.62:42319)
0928 23:17:48.229728 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:48.229729 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:48.229739 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:17:48.229788 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:48.229789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:48.229798 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:48.229851 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:48.230113 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.230605 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.231307 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.231506 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.231608 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:48.233866 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45407: BlockingRecv error: recv got EOF from 127.0.98.1:45407 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
Sep 28 23:17:48 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:48 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:48.356465 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:48.351878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:52635)
0928 23:17:48.352000 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:48.352013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:48.352075 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:48.352199 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:48.352200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:48.352363 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:48.352407 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:48.352409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.353031 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.353032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:48.353824 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:48.353828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.354191 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.354192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:48.354246 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:48.354701 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:48.354707 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:48.356391 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
W20260928 23:17:48.356535 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:48.351979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52635 (local address 127.0.98.62:42319)
0928 23:17:48.352053 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:48.352055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:48.352063 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:48.352135 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:48.352136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:48.352147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:48.352191 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:48.352475 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.352960 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.353892 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.354125 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.354176 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:48.356485 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52635: BlockingRecv error: recv got EOF from 127.0.98.2:52635 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 23:17:48 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:48 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:48.502034 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:48.497413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:40399)
0928 23:17:48.497542 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:48.497553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:48.497605 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:48.497711 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:48.497712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:48.497896 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:48.497941 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:48.497943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.498567 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.498568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:48.499152 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:48.499155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.499689 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.499690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:48.499777 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:48.500230 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:48.500234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:48.501964 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
W20260928 23:17:48.502094 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:48.497483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40399 (local address 127.0.98.62:42319)
0928 23:17:48.497535 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:48.497537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:48.497615 (+ 78us) server_negotiation.cc:407] Connection header received
0928 23:17:48.497640 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:48.497640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:48.497651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:48.497700 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:48.497993 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.498484 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.499255 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:48.499585 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:48.499652 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:48.502042 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40399: BlockingRecv error: recv got EOF from 127.0.98.3:40399 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 23:17:49 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:49 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:49.238510 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:49.234334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:47305)
0928 23:17:49.234468 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:49.234479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:49.234541 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:49.234671 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:49.234672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:49.234827 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:49.234873 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:49.234875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.235569 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.235570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:49.236093 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:49.236094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.236404 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.236404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:49.236442 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:49.236881 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:49.236886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:49.238451 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
W20260928 23:17:49.238623 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:49.234431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47305 (local address 127.0.98.62:42319)
0928 23:17:49.234537 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:49.234538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:49.234547 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:49.234585 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:49.234586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:49.234607 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:49.234662 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:49.234957 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.235485 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.236167 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.236353 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.236399 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:49.238534 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47305: BlockingRecv error: recv got EOF from 127.0.98.1:47305 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
Sep 28 23:17:49 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:49 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:49.360980 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:49.356874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:44507)
0928 23:17:49.357021 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:49.357031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:49.357091 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:49.357201 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:49.357202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:49.357352 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:49.357399 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:49.357400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.357986 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.357987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:49.358496 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:49.358497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.358782 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.358782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:49.358818 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:49.359246 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:49.359250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:49.360921 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
W20260928 23:17:49.361045 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:49.356947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44507 (local address 127.0.98.62:42319)
0928 23:17:49.357009 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:49.357011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:49.357087 (+ 76us) server_negotiation.cc:407] Connection header received
0928 23:17:49.357145 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:49.357146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:49.357155 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:49.357207 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:49.357445 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.357933 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.358552 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.358733 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.358778 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:49.360992 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44507: BlockingRecv error: recv got EOF from 127.0.98.2:44507 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 23:17:49 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:49 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:49.519321 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:49.502500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:59939)
0928 23:17:49.502634 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:49.502644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:49.502711 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:49.502843 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:49.502843 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:49.503037 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:49.503085 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:49.503086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.503756 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.503757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:49.504311 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:49.504313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.504584 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.504585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:49.504625 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:49.517611 (+ 12986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:49.517617 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:49.519262 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
W20260928 23:17:49.519423 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:49.502551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59939 (local address 127.0.98.62:42319)
0928 23:17:49.502603 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:49.502605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:49.502705 (+ 100us) server_negotiation.cc:407] Connection header received
0928 23:17:49.502756 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:49.502757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:49.502768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:49.502815 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:49.503137 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.503689 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.504369 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:49.504543 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:49.504583 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:49.519361 (+ 14778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59939: BlockingRecv error: recv got EOF from 127.0.98.3:59939 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 23:17:50 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:50 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:50.243067 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:50.238989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:58879)
0928 23:17:50.239114 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:50.239124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:50.239171 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:50.239283 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:50.239284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:50.239430 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:50.239474 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:50.239476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.240128 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.240129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:50.240640 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:50.240641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.240944 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.240945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:50.240982 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.241387 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:50.241390 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:50.243009 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
W20260928 23:17:50.243204 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:50.239060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58879 (local address 127.0.98.62:42319)
0928 23:17:50.239154 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:50.239155 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:50.239170 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:17:50.239212 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:50.239213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:50.239222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:50.239279 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:50.239538 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.240071 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.240702 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.240894 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.240940 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.243134 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58879: BlockingRecv error: recv got EOF from 127.0.98.1:58879 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
Sep 28 23:17:50 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:17:50 dist-test-slave-djkr krb5kdc[8515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:17:50.365892 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:50.361436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:37797)
0928 23:17:50.361565 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:50.361576 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:50.361639 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:50.361791 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:50.361791 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:50.361943 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:50.361986 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:50.361988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.362656 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.362657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:50.363193 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:50.363195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.363575 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.363576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:50.363631 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.364015 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:50.364019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:50.365823 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42319: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
W20260928 23:17:50.365952 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:50.361497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37797 (local address 127.0.98.62:42319)
0928 23:17:50.361559 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:50.361561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:50.361628 (+ 67us) server_negotiation.cc:407] Connection header received
0928 23:17:50.361695 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:50.361696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:50.361706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:50.361775 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:50.362055 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.362595 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.363288 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.363492 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.363571 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.365904 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37797: BlockingRecv error: recv got EOF from 127.0.98.2:37797 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:17:50.428557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 9098
I20260928 23:17:50.432458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 9250
I20260928 23:17:50.435873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 9382
I20260928 23:17:50.439283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 9028
2026-09-28T23:17:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17264 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T23:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:50Z Disabled control of system clock
I20260928 23:17:50.463318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37415
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 23:17:50.542716 9580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:50.542896 9580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:50.542914 9580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:50.544055 9580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:50.544092 9580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:50.544103 9580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:50.544114 9580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:50.544131 9580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:50.545392 9580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:50.545565 9580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:50.545776 9580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:50.550416 9588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:50.550454 9580 server_base.cc:1034] running on GCE node
W20260928 23:17:50.550452 9585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:50.550424 9586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:50.550918 9580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:50.551158 9580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:50.552296 9580 hybrid_clock.cc:648] HybridClock initialized: now 1790637470552285 us; error 27 us; skew 500 ppm
I20260928 23:17:50.553433 9580 webserver.cc:460] Webserver started at http://127.0.98.62:41049/ using document root <none> and password file <none>
I20260928 23:17:50.553632 9580 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:50.553675 9580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:50.553788 9580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:50.554813 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "f0304e77e7844e37b5f71f5254e9d336"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "3c5b9c12f2712170ff737e271d4e3363"
server_key_iv: "272617e3bb3d53b7413791d3ca89f58e"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.555126 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0304e77e7844e37b5f71f5254e9d336"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "3c5b9c12f2712170ff737e271d4e3363"
server_key_iv: "272617e3bb3d53b7413791d3ca89f58e"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.556288 9580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:50.556915 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:50.557050 9580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:50.557154 9580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "f0304e77e7844e37b5f71f5254e9d336"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "3c5b9c12f2712170ff737e271d4e3363"
server_key_iv: "272617e3bb3d53b7413791d3ca89f58e"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.557219 9580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:50.574520 9580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:50.574780 9580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:50.574896 9580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:50.578505 9580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37415
I20260928 23:17:50.578578 9646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37415 every 8 connection(s)
I20260928 23:17:50.578841 9580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:50.579416 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:50.581897 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336: Bootstrap starting.
I20260928 23:17:50.582419 9647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:50.582652 9647 log.cc:824] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:50.583250 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336: No bootstrap required, opened a new log
I20260928 23:17:50.584738 9647 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0304e77e7844e37b5f71f5254e9d336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260928 23:17:50.584865 9647 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:50.584885 9647 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0304e77e7844e37b5f71f5254e9d336, State: Initialized, Role: FOLLOWER
I20260928 23:17:50.585013 9647 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0304e77e7844e37b5f71f5254e9d336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260928 23:17:50.585099 9647 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:50.585122 9647 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:50.585167 9647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:50.585628 9647 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0304e77e7844e37b5f71f5254e9d336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260928 23:17:50.585706 9647 leader_election.cc:302] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0304e77e7844e37b5f71f5254e9d336; no voters:
I20260928 23:17:50.585840 9647 leader_election.cc:288] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:50.585880 9650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:50.586027 9650 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [term 1 LEADER]: Becoming Leader. State: Replica: f0304e77e7844e37b5f71f5254e9d336, State: Running, Role: LEADER
I20260928 23:17:50.586061 9647 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:50.586136 9650 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0304e77e7844e37b5f71f5254e9d336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260928 23:17:50.586508 9652 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0304e77e7844e37b5f71f5254e9d336. Latest consensus state: current_term: 1 leader_uuid: "f0304e77e7844e37b5f71f5254e9d336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0304e77e7844e37b5f71f5254e9d336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } } }
I20260928 23:17:50.586596 9652 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:50.586643 9651 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0304e77e7844e37b5f71f5254e9d336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0304e77e7844e37b5f71f5254e9d336" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } } }
I20260928 23:17:50.586714 9651 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:50.586798 9659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:50.587020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 9580
I20260928 23:17:50.587085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:50.587268 392 external_mini_cluster.cc:1428] Setting key 1671b638d85b0b5ad559540d37641949
I20260928 23:17:50.587265 9659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:50.589068 9659 catalog_manager.cc:1347] Generated new cluster ID: 69a00abf4e82424fa1ed581fbf3f89c2
I20260928 23:17:50.589120 9659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:50.592203 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:50.587714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.0.98.62:37415)
0928 23:17:50.588330 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:50.588338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:50.588368 (+ 30us) server_negotiation.cc:407] Connection header received
0928 23:17:50.588483 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:50.588489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:50.588551 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:50.588684 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:50.588873 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.589808 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.590524 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.590926 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.591033 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.591655 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:50.591659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:50.591659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:50.591815 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:50.591844 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:50.591854 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:50.591907 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:50.591981 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 23:17:50.592051 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":52,"threads_started":1}
I20260928 23:17:50.621232 9659 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:50.621910 9659 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:50.631925 9659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0304e77e7844e37b5f71f5254e9d336: Generated new TSK 0
I20260928 23:17:50.632093 9659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:17:50.635352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--builtin_ntp_servers=127.0.98.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 23:17:50.711805 9671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:50.711988 9671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:50.712016 9671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:50.713114 9671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:50.713151 9671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:50.713182 9671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:50.714426 9671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:50.714649 9671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:50.714893 9671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:50.720032 9676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:50.720153 9671 server_base.cc:1034] running on GCE node
W20260928 23:17:50.720055 9677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:50.720057 9679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:50.720446 9671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:50.720655 9671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:50.721817 9671 hybrid_clock.cc:648] HybridClock initialized: now 1790637470721805 us; error 28 us; skew 500 ppm
I20260928 23:17:50.722952 9671 webserver.cc:460] Webserver started at http://127.0.98.1:39121/ using document root <none> and password file <none>
I20260928 23:17:50.723145 9671 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:50.723186 9671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:50.723315 9671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:50.724349 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "85670a29ce9d4c3998932a782cb932ad"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "4da8f5de8228863bbcaccc313f2b188a"
server_key_iv: "584aecd4ed20c22e8fb0d0c25fb3b560"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.724651 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "85670a29ce9d4c3998932a782cb932ad"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "4da8f5de8228863bbcaccc313f2b188a"
server_key_iv: "584aecd4ed20c22e8fb0d0c25fb3b560"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.725821 9671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:17:50.726433 9685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:50.726573 9671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:50.726649 9671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "85670a29ce9d4c3998932a782cb932ad"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "4da8f5de8228863bbcaccc313f2b188a"
server_key_iv: "584aecd4ed20c22e8fb0d0c25fb3b560"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.726715 9671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:50.736645 9671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:50.736881 9671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:50.736990 9671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:50.737166 9671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:50.737452 9671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:50.737483 9671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:50.737509 9671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:50.737531 9671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:50.742776 9671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35619
I20260928 23:17:50.742815 9798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35619 every 8 connection(s)
I20260928 23:17:50.743089 9671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:50.747660 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:50.743753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55671 (local address 127.0.98.62:37415)
0928 23:17:50.743799 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:50.743801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:50.744003 (+ 202us) server_negotiation.cc:407] Connection header received
0928 23:17:50.744491 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:50.744492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:50.744503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:50.744537 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:50.745133 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.745608 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.746499 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.746669 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.746727 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.747448 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:50.747449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:50.747449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:50.747453 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:50.747463 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:50.747466 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:50.747499 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:50.747564 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 23:17:50.747583 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:17:50.747860 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:50.743744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37415 (local address 127.0.98.1:55671)
0928 23:17:50.743895 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:50.743926 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:50.744403 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:50.744596 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:50.744599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:50.744607 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:50.745078 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:50.745082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.745664 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.745665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:50.746443 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:50.746445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.746718 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.746719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:50.746854 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.747342 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:50.747342 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:50.747369 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:50.747391 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:50.747500 (+ 109us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:50.747502 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:50.747505 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:50.747585 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:17:50.747674 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260928 23:17:50.748310 9799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37415
I20260928 23:17:50.748380 9799 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:50.748539 9799 heartbeater.cc:511] Master 127.0.98.62:37415 requested a full tablet report, sending...
I20260928 23:17:50.748996 9611 ts_manager.cc:194] Registered new tserver with Master: 85670a29ce9d4c3998932a782cb932ad (127.0.98.1:35619)
I20260928 23:17:50.749594 9611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55671
I20260928 23:17:50.749701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 9671
I20260928 23:17:50.749785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:50.749972 392 external_mini_cluster.cc:1428] Setting key 6782dff4a802ac119686e61b150132a0
I20260928 23:17:50.750835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--builtin_ntp_servers=127.0.98.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 23:17:50.828667 9802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:50.828857 9802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:50.828877 9802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:50.830096 9802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:50.830196 9802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:50.830247 9802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:50.831537 9802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:50.831736 9802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:50.831961 9802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:50.836575 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:50.836562 9808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:50.836566 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:50.836717 9802 server_base.cc:1034] running on GCE node
I20260928 23:17:50.836915 9802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:50.837126 9802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:50.838264 9802 hybrid_clock.cc:648] HybridClock initialized: now 1790637470838251 us; error 25 us; skew 500 ppm
I20260928 23:17:50.839514 9802 webserver.cc:460] Webserver started at http://127.0.98.2:36423/ using document root <none> and password file <none>
I20260928 23:17:50.839753 9802 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:50.839802 9802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:50.839923 9802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:50.840893 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e07b63ed243495bb8c827ccbc9eff75"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "669a3cdd89db27f7c15688b21362d66e"
server_key_iv: "6b5ec9b58da517117001beb07e5354c6"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.841215 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e07b63ed243495bb8c827ccbc9eff75"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "669a3cdd89db27f7c15688b21362d66e"
server_key_iv: "6b5ec9b58da517117001beb07e5354c6"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.842331 9802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:50.842909 9816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:50.843057 9802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:50.843164 9802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "3e07b63ed243495bb8c827ccbc9eff75"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "669a3cdd89db27f7c15688b21362d66e"
server_key_iv: "6b5ec9b58da517117001beb07e5354c6"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.843230 9802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:50.866858 9802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:50.867137 9802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:50.867270 9802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:50.867472 9802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:50.867805 9802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:50.867846 9802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:50.867875 9802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:50.867895 9802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:50.873045 9802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41347
I20260928 23:17:50.873102 9929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41347 every 8 connection(s)
I20260928 23:17:50.873382 9802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:50.875000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 9802
I20260928 23:17:50.875087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:50.875269 392 external_mini_cluster.cc:1428] Setting key 4cb016f7a3f10dddeb7ca2983948fc44
I20260928 23:17:50.876133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--builtin_ntp_servers=127.0.98.20:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 23:17:50.878432 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:50.873992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50961 (local address 127.0.98.62:37415)
0928 23:17:50.874085 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:50.874087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:50.874280 (+ 193us) server_negotiation.cc:407] Connection header received
0928 23:17:50.874736 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:50.874737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:50.874746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:50.874809 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:50.875633 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.876236 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.877191 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.877404 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.877458 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.878195 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:50.878196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:50.878197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:50.878201 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:50.878215 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:50.878219 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:50.878268 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:50.878333 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 23:17:50.878358 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:17:50.878634 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:50.873965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37415 (local address 127.0.98.2:50961)
0928 23:17:50.874147 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:50.874186 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:50.874654 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:50.874888 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:50.874892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:50.874902 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:50.875531 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:50.875537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.876338 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.876339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:50.877109 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:50.877111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.877476 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.877477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:50.877604 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.878069 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:50.878070 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:50.878097 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:50.878119 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:50.878269 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:50.878274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:50.878276 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:17:50.878338 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 23:17:50.878431 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260928 23:17:50.878985 9930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37415
I20260928 23:17:50.879063 9930 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:50.879222 9930 heartbeater.cc:511] Master 127.0.98.62:37415 requested a full tablet report, sending...
I20260928 23:17:50.879528 9611 ts_manager.cc:194] Registered new tserver with Master: 3e07b63ed243495bb8c827ccbc9eff75 (127.0.98.2:41347)
I20260928 23:17:50.879890 9611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50961
W20260928 23:17:50.953249 9933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:50.953434 9933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:50.953460 9933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:50.954584 9933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:50.954623 9933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:50.954654 9933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:50.955955 9933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:50.956166 9933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:50.956422 9933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:50.960843 9939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:50.960819 9938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:50.960878 9941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:50.960986 9933 server_base.cc:1034] running on GCE node
I20260928 23:17:50.961153 9933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:50.961340 9933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:50.962463 9933 hybrid_clock.cc:648] HybridClock initialized: now 1790637470962450 us; error 28 us; skew 500 ppm
I20260928 23:17:50.963603 9933 webserver.cc:460] Webserver started at http://127.0.98.3:39425/ using document root <none> and password file <none>
I20260928 23:17:50.963831 9933 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:50.963874 9933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:50.963979 9933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:50.964931 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "cd4a94fc029545388594f298ddf60d9a"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "9ffc035aba363fcb42a19741599191b2"
server_key_iv: "40cfb681d2f4053a3438459dfb07060f"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.965243 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cd4a94fc029545388594f298ddf60d9a"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "9ffc035aba363fcb42a19741599191b2"
server_key_iv: "40cfb681d2f4053a3438459dfb07060f"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.966362 9933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:50.967010 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:50.967142 9933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:50.967211 9933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "cd4a94fc029545388594f298ddf60d9a"
format_stamp: "Formatted at 2026-09-28 23:17:50 on dist-test-slave-djkr"
server_key: "9ffc035aba363fcb42a19741599191b2"
server_key_iv: "40cfb681d2f4053a3438459dfb07060f"
server_key_version: "encryptionkey@0"
I20260928 23:17:50.967274 9933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:50.979307 9933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:50.979566 9933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:50.979696 9933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:50.979887 9933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:50.980172 9933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:50.980203 9933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:50.980224 9933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:50.980247 9933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:50.985402 9933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36727
I20260928 23:17:50.985495 10060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36727 every 8 connection(s)
I20260928 23:17:50.985898 9933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:50.990134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 9933
I20260928 23:17:50.990199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:50.990362 392 external_mini_cluster.cc:1428] Setting key b5d62970901c15e1688bbd6b73bbbb98
I20260928 23:17:50.990686 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:50.986550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60261 (local address 127.0.98.62:37415)
0928 23:17:50.986604 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:50.986606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:50.986856 (+ 250us) server_negotiation.cc:407] Connection header received
0928 23:17:50.987342 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:50.987343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:50.987352 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:50.987399 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:50.988023 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.988519 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.989395 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.989570 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.989613 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.990344 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:50.990346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:50.990346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:50.990350 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:50.990365 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:50.990369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:50.990416 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:50.990587 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 23:17:50.990613 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:17:50.990803 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:50.986564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37415 (local address 127.0.98.3:60261)
0928 23:17:50.986728 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:50.986765 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:50.987265 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:50.987453 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:50.987456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:50.987465 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:50.987956 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:50.987961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.988576 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.988577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:50.989335 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:50.989337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:50.989621 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:50.989622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:50.989729 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:50.990206 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:50.990206 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:50.990243 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:50.990268 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:50.990423 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:50.990427 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:50.990430 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:50.990490 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 23:17:50.990590 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260928 23:17:50.991178 10061 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37415
I20260928 23:17:50.991256 10061 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:50.991386 10061 heartbeater.cc:511] Master 127.0.98.62:37415 requested a full tablet report, sending...
I20260928 23:17:50.991701 9610 ts_manager.cc:194] Registered new tserver with Master: cd4a94fc029545388594f298ddf60d9a (127.0.98.3:36727)
I20260928 23:17:50.992015 9610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60261
I20260928 23:17:51.001905 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:17:51.003520 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:51.003086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.0.98.62:37415)
0928 23:17:51.003142 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:51.003144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:51.003233 (+ 89us) server_negotiation.cc:407] Connection header received
0928 23:17:51.003283 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:51.003284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:51.003294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:51.003335 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:51.003464 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54484: BlockingRecv error: recv got EOF from 127.0.0.1:54484 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:17:51.003651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 9671
I20260928 23:17:51.007300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 9802
I20260928 23:17:51.011030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 9933
I20260928 23:17:51.014464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 9580
2026-09-28T23:17:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (574 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T23:17:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:51Z Disabled control of system clock
I20260928 23:17:51.037918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37691
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:17:51.115355 10072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:51.115566 10072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:51.115586 10072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:51.116642 10072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:51.116672 10072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:51.116683 10072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:51.116693 10072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:51.116703 10072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:51.117967 10072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:51.118153 10072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:51.118367 10072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:51.122704 10080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:51.122705 10078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:51.122700 10077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:51.122746 10072 server_base.cc:1034] running on GCE node
I20260928 23:17:51.123133 10072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:51.123394 10072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:51.124521 10072 hybrid_clock.cc:648] HybridClock initialized: now 1790637471124503 us; error 35 us; skew 500 ppm
I20260928 23:17:51.125620 10072 webserver.cc:460] Webserver started at http://127.0.98.62:35633/ using document root <none> and password file <none>
I20260928 23:17:51.125808 10072 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:51.125849 10072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:51.125965 10072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:51.126920 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "bfc9db7fd0704349888e2326a908fde6"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "fc89091c429c0d03f735c050af9d2a56"
server_key_iv: "8302d9d0006b96733160f9f3cad268c3"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.127230 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "bfc9db7fd0704349888e2326a908fde6"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "fc89091c429c0d03f735c050af9d2a56"
server_key_iv: "8302d9d0006b96733160f9f3cad268c3"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.128379 10072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:51.129029 10086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:51.129171 10072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:51.129271 10072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "bfc9db7fd0704349888e2326a908fde6"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "fc89091c429c0d03f735c050af9d2a56"
server_key_iv: "8302d9d0006b96733160f9f3cad268c3"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.129331 10072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:51.132491 10072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:51.132704 10072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:51.132814 10072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:51.136183 10072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37691
I20260928 23:17:51.136237 10138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37691 every 8 connection(s)
I20260928 23:17:51.136499 10072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:51.137050 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:51.139472 10139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6: Bootstrap starting.
I20260928 23:17:51.139984 10139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:51.140209 10139 log.cc:824] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:51.140725 10139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6: No bootstrap required, opened a new log
I20260928 23:17:51.141335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 10072
I20260928 23:17:51.141404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:51.141541 392 external_mini_cluster.cc:1428] Setting key d6a3233668b62729dd1fea7a85b7007c
I20260928 23:17:51.142421 10139 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc9db7fd0704349888e2326a908fde6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } }
I20260928 23:17:51.142555 10139 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:51.142588 10139 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc9db7fd0704349888e2326a908fde6, State: Initialized, Role: FOLLOWER
I20260928 23:17:51.142737 10139 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc9db7fd0704349888e2326a908fde6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } }
I20260928 23:17:51.143112 10139 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:51.143191 10139 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:51.143237 10139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:51.143800 10139 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc9db7fd0704349888e2326a908fde6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } }
I20260928 23:17:51.143884 10139 leader_election.cc:302] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfc9db7fd0704349888e2326a908fde6; no voters:
I20260928 23:17:51.143914 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:51.141965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.0.98.62:37691)
0928 23:17:51.142143 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:51.142149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:51.142172 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:17:51.142392 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:51.142398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:51.142431 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:51.142533 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:51.143158 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:51.143162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:51.143162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:51.143165 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:51.143203 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:51.143216 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:51.143282 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:51.143407 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 23:17:51.143452 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260928 23:17:51.144025 10139 leader_election.cc:288] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:51.144058 10144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:51.144210 10144 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [term 1 LEADER]: Becoming Leader. State: Replica: bfc9db7fd0704349888e2326a908fde6, State: Running, Role: LEADER
I20260928 23:17:51.144248 10139 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:51.144317 10144 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc9db7fd0704349888e2326a908fde6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } }
I20260928 23:17:51.144614 10145 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfc9db7fd0704349888e2326a908fde6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc9db7fd0704349888e2326a908fde6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } } }
I20260928 23:17:51.144678 10145 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:51.144634 10146 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfc9db7fd0704349888e2326a908fde6. Latest consensus state: current_term: 1 leader_uuid: "bfc9db7fd0704349888e2326a908fde6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc9db7fd0704349888e2326a908fde6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37691 } } }
I20260928 23:17:51.144717 10146 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:51.144850 10150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:51.145303 10150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:17:51.146715 10150 catalog_manager.cc:1347] Generated new cluster ID: dbdc930c87bd4c9c8b2180abcae1e3d9
I20260928 23:17:51.146756 10150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:51.156235 10150 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:51.156368 10150 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:51.164494 10150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfc9db7fd0704349888e2326a908fde6: Generated new TSK 0
I20260928 23:17:51.164654 10150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:17:51.167412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--builtin_ntp_servers=127.0.98.20:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:17:51.261941 10163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:51.262123 10163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:51.262144 10163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:51.263268 10163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:51.263303 10163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:51.263327 10163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:51.264647 10163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:51.264856 10163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:51.265144 10163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:51.269730 10171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:51.269812 10169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:51.269820 10168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:51.270073 10163 server_base.cc:1034] running on GCE node
I20260928 23:17:51.270246 10163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:51.270479 10163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:51.271610 10163 hybrid_clock.cc:648] HybridClock initialized: now 1790637471271593 us; error 31 us; skew 500 ppm
I20260928 23:17:51.272912 10163 webserver.cc:460] Webserver started at http://127.0.98.1:38227/ using document root <none> and password file <none>
I20260928 23:17:51.273160 10163 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:51.273217 10163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:51.273343 10163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:51.274725 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "21b4e8196ec247069ed26af48d1ecabd"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "138254b47730abd7aee3cbedfb622f1d"
server_key_iv: "ec47ca5d91b7ae9452d7bd398305a6de"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.275110 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "21b4e8196ec247069ed26af48d1ecabd"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "138254b47730abd7aee3cbedfb622f1d"
server_key_iv: "ec47ca5d91b7ae9452d7bd398305a6de"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.276640 10163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:51.277474 10177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:51.277628 10163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:51.277714 10163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "21b4e8196ec247069ed26af48d1ecabd"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "138254b47730abd7aee3cbedfb622f1d"
server_key_iv: "ec47ca5d91b7ae9452d7bd398305a6de"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.277791 10163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:51.282117 10163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:51.282366 10163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:51.282477 10163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:51.282660 10163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:51.282939 10163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:51.282974 10163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:51.283004 10163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:51.283026 10163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:51.289134 10163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38197
I20260928 23:17:51.289167 10290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38197 every 8 connection(s)
I20260928 23:17:51.289453 10163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:51.292243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 10163
I20260928 23:17:51.292294 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:51.290078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39363 (local address 127.0.98.62:37691)
0928 23:17:51.290135 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:51.290138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:51.290336 (+ 198us) server_negotiation.cc:407] Connection header received
0928 23:17:51.290796 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:51.290797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:51.290803 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:51.290862 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:51.292004 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:51.292005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:51.292006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:51.292010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:51.292027 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:51.292031 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:51.292107 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:51.292204 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 23:17:51.292219 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:17:51.292321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:51.292455 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:51.290063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37691 (local address 127.0.98.1:39363)
0928 23:17:51.290222 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:51.290260 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:51.290695 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:51.290948 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:51.290951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:51.290959 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:51.291868 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:51.291868 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:51.291895 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:51.291921 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:51.292120 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:51.292125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:51.292128 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:51.292209 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 23:17:51.292267 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260928 23:17:51.292486 392 external_mini_cluster.cc:1428] Setting key 39a87e9e5d1a81fd84c9e1c7d1480537
I20260928 23:17:51.293165 10291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37691
I20260928 23:17:51.293256 10291 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:51.293393 10291 heartbeater.cc:511] Master 127.0.98.62:37691 requested a full tablet report, sending...
I20260928 23:17:51.293413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--builtin_ntp_servers=127.0.98.20:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 23:17:51.293794 10102 ts_manager.cc:194] Registered new tserver with Master: 21b4e8196ec247069ed26af48d1ecabd (127.0.98.1:38197)
W20260928 23:17:51.371286 10294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:51.371477 10294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:51.371498 10294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:51.372627 10294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:51.372666 10294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:51.372691 10294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:51.373944 10294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:51.374118 10294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:51.374344 10294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:17:51.379212 10294 server_base.cc:1034] running on GCE node
W20260928 23:17:51.379196 10300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:51.379196 10302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:51.379185 10299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:51.379597 10294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:51.379817 10294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:51.380941 10294 hybrid_clock.cc:648] HybridClock initialized: now 1790637471380929 us; error 28 us; skew 500 ppm
I20260928 23:17:51.382161 10294 webserver.cc:460] Webserver started at http://127.0.98.2:38831/ using document root <none> and password file <none>
I20260928 23:17:51.382387 10294 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:51.382433 10294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:51.382550 10294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:51.383597 10294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "c796829f0e0e4ecda6eacce75c717f06"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "306a03e07ff0784a42123a3bed6184a4"
server_key_iv: "ce983668ebaf782d7441531da3ca6d27"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.383914 10294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c796829f0e0e4ecda6eacce75c717f06"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "306a03e07ff0784a42123a3bed6184a4"
server_key_iv: "ce983668ebaf782d7441531da3ca6d27"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.385035 10294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:17:51.385653 10308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:51.385804 10294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:51.385877 10294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "c796829f0e0e4ecda6eacce75c717f06"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "306a03e07ff0784a42123a3bed6184a4"
server_key_iv: "ce983668ebaf782d7441531da3ca6d27"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.385933 10294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:51.389070 10294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:51.389303 10294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:51.389415 10294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:51.389595 10294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:51.389868 10294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:51.389899 10294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:51.389923 10294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:51.389947 10294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:51.395032 10294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41575
I20260928 23:17:51.395071 10421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41575 every 8 connection(s)
I20260928 23:17:51.395351 10294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:17:51.397351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 10294
I20260928 23:17:51.397442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:51.397594 392 external_mini_cluster.cc:1428] Setting key 1a4029ca55da526068381011c74bae8e
I20260928 23:17:51.398057 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:51.395981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40183 (local address 127.0.98.62:37691)
0928 23:17:51.396034 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:51.396036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:51.396240 (+ 204us) server_negotiation.cc:407] Connection header received
0928 23:17:51.396659 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:51.396660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:51.396665 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:51.396720 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:51.397836 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:51.397838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:51.397839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:51.397843 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:51.397864 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:51.397869 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:51.397921 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:51.397995 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 23:17:51.398006 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:17:51.398253 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:51.395964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37691 (local address 127.0.98.2:40183)
0928 23:17:51.396126 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:51.396164 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:51.396591 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:51.396778 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:51.396781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:51.396789 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:51.397652 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:51.397652 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:51.397689 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:51.397731 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:51.397925 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:51.397930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:51.397932 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:17:51.398001 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:17:51.398057 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260928 23:17:51.398511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--builtin_ntp_servers=127.0.98.20:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 23:17:51.398600 10422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37691
I20260928 23:17:51.398670 10422 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:51.398805 10422 heartbeater.cc:511] Master 127.0.98.62:37691 requested a full tablet report, sending...
I20260928 23:17:51.399137 10102 ts_manager.cc:194] Registered new tserver with Master: c796829f0e0e4ecda6eacce75c717f06 (127.0.98.2:41575)
W20260928 23:17:51.475041 10425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:51.475237 10425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:51.475258 10425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:51.476425 10425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:51.476464 10425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:51.476491 10425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:51.477819 10425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:51.478029 10425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:51.478283 10425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:51.482838 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:51.482854 10433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:51.482829 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:51.482949 10425 server_base.cc:1034] running on GCE node
I20260928 23:17:51.483170 10425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:51.483385 10425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:51.484519 10425 hybrid_clock.cc:648] HybridClock initialized: now 1790637471484491 us; error 42 us; skew 500 ppm
I20260928 23:17:51.485595 10425 webserver.cc:460] Webserver started at http://127.0.98.3:36865/ using document root <none> and password file <none>
I20260928 23:17:51.485819 10425 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:51.485867 10425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:51.485991 10425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:51.486980 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "1c76e9399cf747ca92f6f4961d65db10"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "1dea274a0c92cb17146cfecc116dc186"
server_key_iv: "72f6b2913e542c7886f88e01dc4a9781"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.487298 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1c76e9399cf747ca92f6f4961d65db10"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "1dea274a0c92cb17146cfecc116dc186"
server_key_iv: "72f6b2913e542c7886f88e01dc4a9781"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.488545 10425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:17:51.489151 10439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:51.489349 10425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:17:51.489420 10425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "1c76e9399cf747ca92f6f4961d65db10"
format_stamp: "Formatted at 2026-09-28 23:17:51 on dist-test-slave-djkr"
server_key: "1dea274a0c92cb17146cfecc116dc186"
server_key_iv: "72f6b2913e542c7886f88e01dc4a9781"
server_key_version: "encryptionkey@0"
I20260928 23:17:51.489487 10425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:51.492622 10425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:51.492828 10425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:51.492918 10425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:51.493075 10425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:51.493297 10425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:51.493325 10425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:51.493342 10425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:51.493359 10425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:51.498688 10425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41439
I20260928 23:17:51.498733 10552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41439 every 8 connection(s)
I20260928 23:17:51.499015 10425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:51.501919 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:51.499725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46955 (local address 127.0.98.62:37691)
0928 23:17:51.499803 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:51.499805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:51.500044 (+ 239us) server_negotiation.cc:407] Connection header received
0928 23:17:51.500513 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:51.500514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:51.500524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:51.500570 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:51.501680 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:51.501682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:51.501682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:17:51.501687 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:51.501703 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:17:51.501708 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:51.501758 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:51.501832 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 23:17:51.501849 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 23:17:51.502091 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:51.499721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37691 (local address 127.0.98.3:46955)
0928 23:17:51.499929 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:51.499964 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:51.500418 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:51.500650 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:51.500654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:51.500663 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:51.501534 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:17:51.501535 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:51.501561 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:17:51.501588 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:51.501762 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:51.501766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:51.501770 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:17:51.501838 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:17:51.501896 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260928 23:17:51.502341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 10425
I20260928 23:17:51.502426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:51.502600 392 external_mini_cluster.cc:1428] Setting key 37c00d6026b8e13d3e46d4e63b47ebac
I20260928 23:17:51.502588 10553 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37691
I20260928 23:17:51.502671 10553 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:51.502924 10553 heartbeater.cc:511] Master 127.0.98.62:37691 requested a full tablet report, sending...
I20260928 23:17:51.503264 10103 ts_manager.cc:194] Registered new tserver with Master: 1c76e9399cf747ca92f6f4961d65db10 (127.0.98.3:41439)
I20260928 23:17:51.514148 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:17:51.515818 10143 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:17:51.515388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.0.98.62:37691)
0928 23:17:51.515452 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:51.515455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:51.515508 (+ 53us) server_negotiation.cc:407] Connection header received
0928 23:17:51.515599 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:51.515600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:51.515605 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:51.515647 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:51.515760 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55280: BlockingRecv error: recv got EOF from 127.0.0.1:55280 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:17:51.515897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 10163
I20260928 23:17:51.519647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 10294
I20260928 23:17:51.523100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 10425
I20260928 23:17:51.527135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 10072
2026-09-28T23:17:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (513 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:51 dist-test-slave-djkr krb5kdc[10563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:51 dist-test-slave-djkr krb5kdc[10563](info): set up 2 sockets
Sep 28 23:17:51 dist-test-slave-djkr krb5kdc[10563](info): commencing operation
krb5kdc: starting...
W20260928 23:17:52.475988 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:52 dist-test-slave-djkr krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:17:52.586984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:52.663730 10576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:52.663934 10576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:52.663964 10576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:52.665167 10576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:17:52.665210 10576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:52.665228 10576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:52.665244 10576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:17:52.665259 10576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:17:52.666678 10576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:52.666903 10576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:52.667095 10576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:52.671528 10581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:52.671567 10582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:52.671566 10584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:52.671710 10576 server_base.cc:1034] running on GCE node
I20260928 23:17:52.672015 10576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:52.672303 10576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:52.673452 10576 hybrid_clock.cc:648] HybridClock initialized: now 1790637472673432 us; error 32 us; skew 500 ppm
Sep 28 23:17:52 dist-test-slave-djkr krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:52.675783 10576 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:17:52.676164 10576 webserver.cc:460] Webserver started at http://127.0.98.62:40603/ using document root <none> and password file <none>
I20260928 23:17:52.676328 10576 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:52.676368 10576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:52.676450 10576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:52.677419 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "16ee68c61f594510a86eb669b8050d29"
format_stamp: "Formatted at 2026-09-28 23:17:52 on dist-test-slave-djkr"
server_key: "c421588b9ed7a61084f85f59b3b438c4"
server_key_iv: "8b69ce4b6b5988164603408db0af6a6c"
server_key_version: "encryptionkey@0"
I20260928 23:17:52.677698 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "16ee68c61f594510a86eb669b8050d29"
format_stamp: "Formatted at 2026-09-28 23:17:52 on dist-test-slave-djkr"
server_key: "c421588b9ed7a61084f85f59b3b438c4"
server_key_iv: "8b69ce4b6b5988164603408db0af6a6c"
server_key_version: "encryptionkey@0"
I20260928 23:17:52.678864 10576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:52.679589 10591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:52.679750 10576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:17:52.679828 10576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "16ee68c61f594510a86eb669b8050d29"
format_stamp: "Formatted at 2026-09-28 23:17:52 on dist-test-slave-djkr"
server_key: "c421588b9ed7a61084f85f59b3b438c4"
server_key_iv: "8b69ce4b6b5988164603408db0af6a6c"
server_key_version: "encryptionkey@0"
I20260928 23:17:52.679888 10576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:52.701610 10576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:52.703799 10576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:52.703935 10576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:52.707420 10576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43377
I20260928 23:17:52.707459 10643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43377 every 8 connection(s)
I20260928 23:17:52.707800 10576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:17:52.708710 10644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:52.710866 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29: Bootstrap starting.
I20260928 23:17:52.711105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 10576
I20260928 23:17:52.711196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:17:52.711372 392 external_mini_cluster.cc:1428] Setting key ee0b72a1b4fd8c3aaed27573999e12ee
I20260928 23:17:52.711532 10644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:52.711869 10644 log.cc:824] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:52.712589 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29: No bootstrap required, opened a new log
I20260928 23:17:52.714676 10644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee68c61f594510a86eb669b8050d29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260928 23:17:52.714838 10644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:52.714871 10644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ee68c61f594510a86eb669b8050d29, State: Initialized, Role: FOLLOWER
I20260928 23:17:52.715029 10644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee68c61f594510a86eb669b8050d29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260928 23:17:52.715108 10644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:17:52.715137 10644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:17:52.715191 10644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:52.715744 10644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee68c61f594510a86eb669b8050d29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260928 23:17:52.715843 10644 leader_election.cc:302] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16ee68c61f594510a86eb669b8050d29; no voters:
I20260928 23:17:52.715986 10644 leader_election.cc:288] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:17:52.716024 10649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:52.716185 10649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [term 1 LEADER]: Becoming Leader. State: Replica: 16ee68c61f594510a86eb669b8050d29, State: Running, Role: LEADER
I20260928 23:17:52.716214 10644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:17:52.716323 10649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee68c61f594510a86eb669b8050d29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260928 23:17:52.716816 10651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16ee68c61f594510a86eb669b8050d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee68c61f594510a86eb669b8050d29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } }
I20260928 23:17:52.716892 10651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:52.717087 10657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:17:52.717118 10650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ee68c61f594510a86eb669b8050d29. Latest consensus state: current_term: 1 leader_uuid: "16ee68c61f594510a86eb669b8050d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ee68c61f594510a86eb669b8050d29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } }
I20260928 23:17:52.717252 10650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29 [sys.catalog]: This master's current role is: LEADER
I20260928 23:17:52.717571 10657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 23:17:52 dist-test-slave-djkr krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:52.719498 10657 catalog_manager.cc:1347] Generated new cluster ID: acdeb4085f044e9b83e49688c36a4306
I20260928 23:17:52.719568 10657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:17:52.722443 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:52.711876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.0.98.62:43377)
0928 23:17:52.712135 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:52.712144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:52.712169 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:17:52.712689 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:52.712699 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:52.712763 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:52.712910 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:52.713445 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:52.714824 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:52.715690 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:52.716287 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:52.716391 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:52.718970 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:52.718974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:52.718975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:52.718987 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:52.720832 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:52.721171 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:52.721173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:52.721174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:52.721260 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:52.721411 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:52.721412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:52.721412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:52.721671 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:52.721790 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:52.722148 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 23:17:52.722263 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
I20260928 23:17:52.744195 10657 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:17:52.744648 10657 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:17:52.749215 10657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29: Generated new TSK 0
I20260928 23:17:52.749356 10657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:17:52.780541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:52.879971 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:52.880162 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:52.880183 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:52.881269 10670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:52.881307 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:52.881335 10670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:17:52.882730 10670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:52.882932 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:52.883137 10670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:17:52.887722 10670 server_base.cc:1034] running on GCE node
W20260928 23:17:52.887686 10678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:52.887696 10676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:52.887686 10675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:52.888046 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:52.888303 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:52.889443 10670 hybrid_clock.cc:648] HybridClock initialized: now 1790637472889431 us; error 28 us; skew 500 ppm
Sep 28 23:17:52 dist-test-slave-djkr krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:52.891739 10670 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:17:52.892133 10670 webserver.cc:460] Webserver started at http://127.0.98.1:40693/ using document root <none> and password file <none>
I20260928 23:17:52.892318 10670 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:52.892362 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:52.892474 10670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:52.893448 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f"
format_stamp: "Formatted at 2026-09-28 23:17:52 on dist-test-slave-djkr"
server_key: "6f6ab0957e60dbca04a5645c6fc9485b"
server_key_iv: "7aa18298fc91fe179a704ddb740dc0b5"
server_key_version: "encryptionkey@0"
I20260928 23:17:52.893749 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f"
format_stamp: "Formatted at 2026-09-28 23:17:52 on dist-test-slave-djkr"
server_key: "6f6ab0957e60dbca04a5645c6fc9485b"
server_key_iv: "7aa18298fc91fe179a704ddb740dc0b5"
server_key_version: "encryptionkey@0"
I20260928 23:17:52.894970 10670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:17:52.895704 10685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:52.895844 10670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:52.895918 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f"
format_stamp: "Formatted at 2026-09-28 23:17:52 on dist-test-slave-djkr"
server_key: "6f6ab0957e60dbca04a5645c6fc9485b"
server_key_iv: "7aa18298fc91fe179a704ddb740dc0b5"
server_key_version: "encryptionkey@0"
I20260928 23:17:52.895977 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:52.913437 10670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:52.915167 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:52.915288 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:52.915470 10670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:52.915812 10670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:52.915858 10670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:52.915885 10670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:52.915908 10670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:52.921290 10670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42365
I20260928 23:17:52.921324 10798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42365 every 8 connection(s)
I20260928 23:17:52.921599 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:17:52.925151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 10670
I20260928 23:17:52.925237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:17:52.925390 392 external_mini_cluster.cc:1428] Setting key 45409abf544af1e02e8f4e7645e36271
Sep 28 23:17:52 dist-test-slave-djkr krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:52.930506 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:52.922209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40695 (local address 127.0.98.62:43377)
0928 23:17:52.922274 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:52.922277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:52.922468 (+ 191us) server_negotiation.cc:407] Connection header received
0928 23:17:52.922945 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:52.922946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:52.922956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:52.923005 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:52.923951 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:52.924449 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:52.925416 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:52.925667 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:52.925733 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:52.927900 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:52.927902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:52.927902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:52.927908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:52.929153 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:52.929606 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:52.929607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:52.929607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:52.929660 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:52.929895 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:52.929896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:52.929896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:52.929990 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:52.930038 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:52.930360 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 23:17:52.930443 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 23:17:52.930688 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:52.922172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43377 (local address 127.0.98.1:40695)
0928 23:17:52.922350 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:52.922384 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:52.922857 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:52.923063 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:52.923066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:52.923687 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:52.923863 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:52.923868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:52.924513 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:52.924514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:52.925336 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:52.925338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:52.925746 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:52.925747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:52.925841 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:52.926280 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:52.926287 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:52.927783 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:52.929238 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:52.929245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:52.929247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:52.929523 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:52.929716 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:52.929716 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:52.929717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:52.929841 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:52.930047 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:52.930052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:52.930204 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 23:17:52.930318 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:17:52.930467 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260928 23:17:52.931180 10799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43377
I20260928 23:17:52.931263 10799 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:52.931417 10799 heartbeater.cc:511] Master 127.0.98.62:43377 requested a full tablet report, sending...
I20260928 23:17:52.931895 10608 ts_manager.cc:194] Registered new tserver with Master: 5ab85aa60faa438aaa4d8bd6ce398e7f (127.0.98.1:42365)
I20260928 23:17:52.932567 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40695
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:17:52.950016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:53.026568 10804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:53.026754 10804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:53.026784 10804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:53.027927 10804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:53.027967 10804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:53.028000 10804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:17:53.029395 10804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:53.029608 10804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:53.029803 10804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:53.034614 10810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:53.034619 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:53.034677 10804 server_base.cc:1034] running on GCE node
W20260928 23:17:53.034623 10809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:53.034965 10804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:53.035164 10804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:53.036322 10804 hybrid_clock.cc:648] HybridClock initialized: now 1790637473036308 us; error 31 us; skew 500 ppm
Sep 28 23:17:53 dist-test-slave-djkr krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:53.038698 10804 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:17:53.039072 10804 webserver.cc:460] Webserver started at http://127.0.98.2:41009/ using document root <none> and password file <none>
I20260928 23:17:53.039255 10804 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:53.039300 10804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:53.039419 10804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:53.040431 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "a9cf65412beb4870816ee04da7a130a2"
format_stamp: "Formatted at 2026-09-28 23:17:53 on dist-test-slave-djkr"
server_key: "0d66c13a02b6a1c545bfde8b3f162d1e"
server_key_iv: "5af7bf574ecb2dc5b7c7896869218824"
server_key_version: "encryptionkey@0"
I20260928 23:17:53.040751 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a9cf65412beb4870816ee04da7a130a2"
format_stamp: "Formatted at 2026-09-28 23:17:53 on dist-test-slave-djkr"
server_key: "0d66c13a02b6a1c545bfde8b3f162d1e"
server_key_iv: "5af7bf574ecb2dc5b7c7896869218824"
server_key_version: "encryptionkey@0"
I20260928 23:17:53.041929 10804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:53.042514 10819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:53.042686 10804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:53.042762 10804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "a9cf65412beb4870816ee04da7a130a2"
format_stamp: "Formatted at 2026-09-28 23:17:53 on dist-test-slave-djkr"
server_key: "0d66c13a02b6a1c545bfde8b3f162d1e"
server_key_iv: "5af7bf574ecb2dc5b7c7896869218824"
server_key_version: "encryptionkey@0"
I20260928 23:17:53.042825 10804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:53.048928 10804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:53.050482 10804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:53.050601 10804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:53.050792 10804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:53.051039 10804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:53.051069 10804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:53.051095 10804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:53.051112 10804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:53.056365 10804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33963
I20260928 23:17:53.056419 10932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33963 every 8 connection(s)
I20260928 23:17:53.056674 10804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:17:53 dist-test-slave-djkr krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:53.064060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 10804
I20260928 23:17:53.064177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:17:53.064352 392 external_mini_cluster.cc:1428] Setting key 274ceb10289c8bef6f95f4a1153c0734
I20260928 23:17:53.064738 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.057285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39633 (local address 127.0.98.62:43377)
0928 23:17:53.057356 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:53.057359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:53.057524 (+ 165us) server_negotiation.cc:407] Connection header received
0928 23:17:53.057944 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:53.057945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:53.057954 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:53.058002 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:53.058806 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.059285 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.060249 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.060449 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.060498 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.062329 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:53.062330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:53.062331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:53.062337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:53.063524 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:53.063878 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:53.063879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:53.063880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:53.063927 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:53.064127 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:53.064128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:53.064129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:53.064248 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:53.064307 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:53.064567 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 23:17:53.064640 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:17:53.064874 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.057274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43377 (local address 127.0.98.2:39633)
0928 23:17:53.057424 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:53.057454 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:53.057875 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:53.058051 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:53.058054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:53.058573 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:53.058743 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.058747 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.059355 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.059356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.060179 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.060181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.060504 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.060505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.060589 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.060961 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:53.060968 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:53.062243 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:53.063605 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:53.063612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:53.063614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:53.063816 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:53.063980 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:53.063981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:53.063981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:53.064075 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:53.064299 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:53.064304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:53.064442 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 23:17:53.064539 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 23:17:53.064678 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260928 23:17:53.065236 10933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43377
I20260928 23:17:53.065326 10933 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:53.065487 10933 heartbeater.cc:511] Master 127.0.98.62:43377 requested a full tablet report, sending...
I20260928 23:17:53.065801 10608 ts_manager.cc:194] Registered new tserver with Master: a9cf65412beb4870816ee04da7a130a2 (127.0.98.2:33963)
I20260928 23:17:53.066145 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39633
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:17:53.086952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:17:53.166354 10938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:17:53.166543 10938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:17:53.166564 10938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:17:53.167680 10938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:17:53.167718 10938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:17:53.167745 10938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:17:53.169144 10938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:17:53.169328 10938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:17:53.169494 10938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:17:53.174070 10946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:53.174077 10944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:17:53.174078 10943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:17:53.174185 10938 server_base.cc:1034] running on GCE node
I20260928 23:17:53.174444 10938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:17:53.174667 10938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:17:53.175798 10938 hybrid_clock.cc:648] HybridClock initialized: now 1790637473175785 us; error 28 us; skew 500 ppm
Sep 28 23:17:53 dist-test-slave-djkr krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:17:53.178056 10938 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:17:53.178488 10938 webserver.cc:460] Webserver started at http://127.0.98.3:40731/ using document root <none> and password file <none>
I20260928 23:17:53.178678 10938 fs_manager.cc:359] Metadata directory not provided
I20260928 23:17:53.178722 10938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:17:53.178849 10938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:17:53.179894 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "8b5b00a8c8974a838ea32cd645faf629"
format_stamp: "Formatted at 2026-09-28 23:17:53 on dist-test-slave-djkr"
server_key: "507d803c3a31674fa8c5a632c70a947d"
server_key_iv: "88f9dcc4d5a863732eb3aa645a689486"
server_key_version: "encryptionkey@0"
I20260928 23:17:53.180207 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8b5b00a8c8974a838ea32cd645faf629"
format_stamp: "Formatted at 2026-09-28 23:17:53 on dist-test-slave-djkr"
server_key: "507d803c3a31674fa8c5a632c70a947d"
server_key_iv: "88f9dcc4d5a863732eb3aa645a689486"
server_key_version: "encryptionkey@0"
I20260928 23:17:53.181396 10938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:17:53.182063 10953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:53.182227 10938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:17:53.182310 10938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "8b5b00a8c8974a838ea32cd645faf629"
format_stamp: "Formatted at 2026-09-28 23:17:53 on dist-test-slave-djkr"
server_key: "507d803c3a31674fa8c5a632c70a947d"
server_key_iv: "88f9dcc4d5a863732eb3aa645a689486"
server_key_version: "encryptionkey@0"
I20260928 23:17:53.182375 10938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:17:53.191033 10938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:17:53.192847 10938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:17:53.192966 10938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:17:53.193140 10938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:17:53.193395 10938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:17:53.193425 10938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:53.193451 10938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:17:53.193475 10938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:17:53.198724 10938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34073
I20260928 23:17:53.198758 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34073 every 8 connection(s)
I20260928 23:17:53.199041 10938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:17:53.200882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 10938
I20260928 23:17:53.200961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:17:53.201117 392 external_mini_cluster.cc:1428] Setting key 7a57aa16101b4d6582ef8c18ed20be57
Sep 28 23:17:53 dist-test-slave-djkr krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:17:53.207463 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.199729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34145 (local address 127.0.98.62:43377)
0928 23:17:53.199784 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:53.199787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:53.199985 (+ 198us) server_negotiation.cc:407] Connection header received
0928 23:17:53.200419 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:53.200420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:53.200429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:53.200474 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:53.201460 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.202068 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.203049 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.203256 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.203310 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.205184 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:53.205185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:53.205185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:53.205192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:53.206276 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:53.206610 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:53.206611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:53.206612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:53.206666 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:53.206872 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:53.206873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:53.206873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:53.206979 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:53.207034 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:53.207299 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 23:17:53.207376 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:17:53.207605 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.199723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43377 (local address 127.0.98.3:34145)
0928 23:17:53.199876 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:53.199909 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:53.200345 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:53.200532 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:53.200534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:53.201119 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:17:53.201373 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.201380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.202147 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.202148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.202968 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.202970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.203322 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.203323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.203450 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.203864 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:17:53.203872 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:17:53.205103 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:17:53.206346 (+ 1243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:53.206352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:53.206354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:53.206549 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:53.206718 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:17:53.206718 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:17:53.206719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:17:53.206819 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:17:53.207031 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:17:53.207036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:17:53.207162 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 23:17:53.207244 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:17:53.207386 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260928 23:17:53.208066 11067 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43377
I20260928 23:17:53.208139 11067 heartbeater.cc:464] Registering TS with master...
I20260928 23:17:53.208300 11067 heartbeater.cc:511] Master 127.0.98.62:43377 requested a full tablet report, sending...
I20260928 23:17:53.208619 10608 ts_manager.cc:194] Registered new tserver with Master: 8b5b00a8c8974a838ea32cd645faf629 (127.0.98.3:34073)
I20260928 23:17:53.209013 10608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34145
I20260928 23:17:53.212625 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:17:53.219389 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.213849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.0.98.62:43377)
0928 23:17:53.213897 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:53.213899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:53.214005 (+ 106us) server_negotiation.cc:407] Connection header received
0928 23:17:53.214043 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:53.214044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:53.214057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:53.214099 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:17:53.214627 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.215101 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.215913 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.216123 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.216197 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.217359 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:17:53.217360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:17:53.217361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:17:53.217367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:17:53.218506 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:53.218805 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:53.218806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:53.218806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:53.218860 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:17:53.218995 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:17:53.218996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:17:53.218996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:17:53.219122 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:17:53.219191 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:53.219262 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 23:17:53.219321 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:17:53.220863 10608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:17:53.222366 10608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:17:53.228212 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.224505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42365 (local address 127.0.0.1:40352)
0928 23:17:53.224690 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:53.224698 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:53.224736 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:53.225083 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:53.225084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:53.225086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:53.225119 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.225120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.226390 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.226391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.227257 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.227259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.227943 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.227944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.228019 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.228032 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:17:53.228093 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 23:17:53.228143 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":32,"thread_start_us":63,"threads_started":1}
I20260928 23:17:53.228369 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.224462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.0.98.3:34073)
0928 23:17:53.224862 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:53.224866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:53.224874 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:53.224907 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:53.224909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:53.224921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:53.225160 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:53.225227 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.225903 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.227249 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.227963 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.228079 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.228136 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:53.228239 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 23:17:53.228290 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":75,"threads_started":1}
I20260928 23:17:53.228300 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.224303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34073 (local address 127.0.0.1:36778)
0928 23:17:53.224624 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:53.224671 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:53.224735 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:53.225024 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:53.225025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:53.225028 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:53.225092 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.225093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.225992 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.225993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.227160 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.227163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.228102 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.228103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.228165 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.228168 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:53.228217 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:17:53.228249 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":62,"threads_started":1}
I20260928 23:17:53.228595 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.224337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33963 (local address 127.0.0.1:40164)
0928 23:17:53.224625 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:53.224674 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:53.224736 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:53.224969 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:53.224972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:53.224974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:53.225034 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.225036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.226377 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.226378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.227457 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.227459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.228376 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.228377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.228441 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.228444 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:17:53.228493 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:17:53.228528 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":32,"thread_start_us":48,"threads_started":1}
I20260928 23:17:53.228816 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.224709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.0.98.1:42365)
0928 23:17:53.224855 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:53.224859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:53.224868 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:17:53.224901 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:53.224903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:53.224914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:53.224980 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:53.225228 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.226297 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.227325 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.227872 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.227935 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.227992 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:53.228688 (+ 696us) server_negotiation.cc:299] Negotiation successful
0928 23:17:53.228734 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260928 23:17:53.228874 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.224443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.0.98.2:33963)
0928 23:17:53.224601 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:53.224606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:53.224813 (+ 207us) server_negotiation.cc:407] Connection header received
0928 23:17:53.224858 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:53.224861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:53.224878 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:53.224951 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:53.225129 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.226275 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.227537 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.228238 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.228295 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.228343 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:53.228760 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 23:17:53.228802 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260928 23:17:53.229184 11001 tablet_service.cc:1467] Processing CreateTablet for tablet 4930ae4ca4094c6da163f73ac7a3f25c (DEFAULT_TABLE table=test-table [id=092632856a9d4873ba98be712ae488d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:53.229489 11001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4930ae4ca4094c6da163f73ac7a3f25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:53.229441 10867 tablet_service.cc:1467] Processing CreateTablet for tablet 4930ae4ca4094c6da163f73ac7a3f25c (DEFAULT_TABLE table=test-table [id=092632856a9d4873ba98be712ae488d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:53.229770 10867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4930ae4ca4094c6da163f73ac7a3f25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:53.231863 11081 tablet_bootstrap.cc:492] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: Bootstrap starting.
I20260928 23:17:53.232254 10733 tablet_service.cc:1467] Processing CreateTablet for tablet 4930ae4ca4094c6da163f73ac7a3f25c (DEFAULT_TABLE table=test-table [id=092632856a9d4873ba98be712ae488d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:17:53.232611 10733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4930ae4ca4094c6da163f73ac7a3f25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:17:53.232681 11081 tablet_bootstrap.cc:654] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:53.233220 11081 log.cc:824] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:53.233923 11081 tablet_bootstrap.cc:492] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: No bootstrap required, opened a new log
I20260928 23:17:53.233997 11081 ts_tablet_manager.cc:1397] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:53.235680 11084 tablet_bootstrap.cc:492] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f: Bootstrap starting.
I20260928 23:17:53.235885 11082 tablet_bootstrap.cc:492] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: Bootstrap starting.
I20260928 23:17:53.235891 11081 raft_consensus.cc:348] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5b00a8c8974a838ea32cd645faf629" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 } } peers { permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 } }
I20260928 23:17:53.236066 11081 raft_consensus.cc:374] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:53.236109 11081 raft_consensus.cc:729] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5b00a8c8974a838ea32cd645faf629, State: Initialized, Role: FOLLOWER
I20260928 23:17:53.236227 11081 consensus_queue.cc:260] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5b00a8c8974a838ea32cd645faf629" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 } } peers { permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 } }
I20260928 23:17:53.236325 11084 tablet_bootstrap.cc:654] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:53.236481 11081 ts_tablet_manager.cc:1428] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:17:53.236519 11082 tablet_bootstrap.cc:654] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: Neither blocks nor log segments found. Creating new log.
I20260928 23:17:53.236639 11067 heartbeater.cc:503] Master 127.0.98.62:43377 was elected leader, sending a full tablet report...
I20260928 23:17:53.236822 11084 log.cc:824] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:53.236974 11082 log.cc:824] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:17:53.237506 11084 tablet_bootstrap.cc:492] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f: No bootstrap required, opened a new log
I20260928 23:17:53.237576 11084 ts_tablet_manager.cc:1397] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:17:53.237785 11082 tablet_bootstrap.cc:492] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: No bootstrap required, opened a new log
I20260928 23:17:53.237849 11082 ts_tablet_manager.cc:1397] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:17:53.238936 11084 raft_consensus.cc:348] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5b00a8c8974a838ea32cd645faf629" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 } } peers { permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 } }
I20260928 23:17:53.239073 11084 raft_consensus.cc:374] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:53.239105 11084 raft_consensus.cc:729] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab85aa60faa438aaa4d8bd6ce398e7f, State: Initialized, Role: FOLLOWER
I20260928 23:17:53.239111 11082 raft_consensus.cc:348] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5b00a8c8974a838ea32cd645faf629" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 } } peers { permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 } }
I20260928 23:17:53.239212 11084 consensus_queue.cc:260] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5b00a8c8974a838ea32cd645faf629" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 } } peers { permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 } }
I20260928 23:17:53.239271 11082 raft_consensus.cc:374] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:17:53.239298 11082 raft_consensus.cc:729] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9cf65412beb4870816ee04da7a130a2, State: Initialized, Role: FOLLOWER
I20260928 23:17:53.239403 11082 consensus_queue.cc:260] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5b00a8c8974a838ea32cd645faf629" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 } } peers { permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 } }
I20260928 23:17:53.239450 11084 ts_tablet_manager.cc:1428] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:17:53.239477 10799 heartbeater.cc:503] Master 127.0.98.62:43377 was elected leader, sending a full tablet report...
I20260928 23:17:53.239658 11082 ts_tablet_manager.cc:1428] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:17:53.239712 10933 heartbeater.cc:503] Master 127.0.98.62:43377 was elected leader, sending a full tablet report...
W20260928 23:17:53.307226 10934 tablet.cc:2369] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:17:53.422246 10800 tablet.cc:2369] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:17:53.449527 11068 tablet.cc:2369] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:17:53.573153 11085 raft_consensus.cc:482] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:17:53.573282 11085 raft_consensus.cc:504] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5b00a8c8974a838ea32cd645faf629" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 } } peers { permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 } }
I20260928 23:17:53.573594 11085 leader_election.cc:288] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9cf65412beb4870816ee04da7a130a2 (127.0.98.2:33963), 5ab85aa60faa438aaa4d8bd6ce398e7f (127.0.98.1:42365)
I20260928 23:17:53.577337 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.573778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42365 (local address 127.0.98.3:56807)
0928 23:17:53.573840 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:53.573854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:53.573904 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:53.574037 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:53.574038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:53.574040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:53.574109 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.574111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.574994 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.574995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.576280 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.576283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.577029 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.577030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.577145 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.577147 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:17:53.577196 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:17:53.577241 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":33}
I20260928 23:17:53.577342 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.573784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33963 (local address 127.0.98.3:36361)
0928 23:17:53.573982 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:53.573990 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:17:53.574047 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:17:53.574166 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:17:53.574166 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:17:53.574167 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:17:53.574241 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.574243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.575054 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.575054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.576293 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:17:53.576294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.577029 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.577030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:17:53.577127 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.577134 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:17:53.577193 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:17:53.577240 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":90,"thread_start_us":89,"threads_started":1}
I20260928 23:17:53.577365 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.573864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36361 (local address 127.0.98.2:33963)
0928 23:17:53.573908 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:53.573911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:53.574049 (+ 138us) server_negotiation.cc:407] Connection header received
0928 23:17:53.574100 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:53.574101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:53.574112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:53.574159 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:53.574301 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.574978 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.576380 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.576931 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.577006 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.577017 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:53.577249 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 23:17:53.577295 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:17:53.577365 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.573852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56807 (local address 127.0.98.1:42365)
0928 23:17:53.573900 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:53.573902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:53.573913 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:17:53.573965 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:53.573966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:53.573976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:53.574025 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:17:53.574166 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.574898 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.576380 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.576938 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.577006 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.577016 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:53.577247 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 23:17:53.577294 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:17:53.577721 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4930ae4ca4094c6da163f73ac7a3f25c" candidate_uuid: "8b5b00a8c8974a838ea32cd645faf629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" is_pre_election: true
I20260928 23:17:53.577744 10887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4930ae4ca4094c6da163f73ac7a3f25c" candidate_uuid: "8b5b00a8c8974a838ea32cd645faf629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9cf65412beb4870816ee04da7a130a2" is_pre_election: true
I20260928 23:17:53.577893 10753 raft_consensus.cc:2454] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b5b00a8c8974a838ea32cd645faf629 in term 0.
I20260928 23:17:53.577893 10887 raft_consensus.cc:2454] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b5b00a8c8974a838ea32cd645faf629 in term 0.
I20260928 23:17:53.578121 10954 leader_election.cc:302] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5b00a8c8974a838ea32cd645faf629, a9cf65412beb4870816ee04da7a130a2; no voters:
I20260928 23:17:53.578315 11085 raft_consensus.cc:2781] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:17:53.578384 11085 raft_consensus.cc:482] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:17:53.578415 11085 raft_consensus.cc:3037] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:53.579128 11085 raft_consensus.cc:504] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5b00a8c8974a838ea32cd645faf629" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 } } peers { permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 } }
I20260928 23:17:53.579246 11085 leader_election.cc:288] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [CANDIDATE]: Term 1 election: Requested vote from peers a9cf65412beb4870816ee04da7a130a2 (127.0.98.2:33963), 5ab85aa60faa438aaa4d8bd6ce398e7f (127.0.98.1:42365)
I20260928 23:17:53.579404 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4930ae4ca4094c6da163f73ac7a3f25c" candidate_uuid: "8b5b00a8c8974a838ea32cd645faf629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f"
I20260928 23:17:53.579416 10887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4930ae4ca4094c6da163f73ac7a3f25c" candidate_uuid: "8b5b00a8c8974a838ea32cd645faf629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9cf65412beb4870816ee04da7a130a2"
I20260928 23:17:53.579486 10887 raft_consensus.cc:3037] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:53.579486 10753 raft_consensus.cc:3037] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:17:53.580343 10887 raft_consensus.cc:2454] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b5b00a8c8974a838ea32cd645faf629 in term 1.
I20260928 23:17:53.580358 10753 raft_consensus.cc:2454] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b5b00a8c8974a838ea32cd645faf629 in term 1.
I20260928 23:17:53.580523 10954 leader_election.cc:302] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5b00a8c8974a838ea32cd645faf629, a9cf65412beb4870816ee04da7a130a2; no voters:
I20260928 23:17:53.580590 11085 raft_consensus.cc:2781] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:17:53.580747 11085 raft_consensus.cc:686] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 1 LEADER]: Becoming Leader. State: Replica: 8b5b00a8c8974a838ea32cd645faf629, State: Running, Role: LEADER
I20260928 23:17:53.580838 11085 consensus_queue.cc:237] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5b00a8c8974a838ea32cd645faf629" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } } peers { permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 } } peers { permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 } }
I20260928 23:17:53.581692 10607 catalog_manager.cc:5495] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b5b00a8c8974a838ea32cd645faf629 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8b5b00a8c8974a838ea32cd645faf629" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5b00a8c8974a838ea32cd645faf629" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 } health_report { overall_health: UNKNOWN } } }
I20260928 23:17:53.666469 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.664153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.0.98.3:34073)
0928 23:17:53.664221 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 23:17:53.664223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:17:53.664231 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:17:53.664272 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:17:53.664273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:17:53.664297 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:17:53.664359 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:17:53.664488 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.664966 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.665731 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:17:53.665926 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:17:53.665979 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:17:53.666134 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:17:53.666239 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:17:53.666311 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:17:53.666349 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:17:53.666371 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:17:53.668216 10753 raft_consensus.cc:1261] T 4930ae4ca4094c6da163f73ac7a3f25c P 5ab85aa60faa438aaa4d8bd6ce398e7f [term 1 FOLLOWER]: Refusing update from remote peer 8b5b00a8c8974a838ea32cd645faf629: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:53.668217 10887 raft_consensus.cc:1261] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2 [term 1 FOLLOWER]: Refusing update from remote peer 8b5b00a8c8974a838ea32cd645faf629: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:17:53.668527 11085 consensus_queue.cc:1035] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ab85aa60faa438aaa4d8bd6ce398e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:53.668669 11091 consensus_queue.cc:1035] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9cf65412beb4870816ee04da7a130a2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:17:53.670816 11094 mvcc.cc:205] Tried to move back new op lower bound from 7334451092142313472 to 7334451091787624448. Current Snapshot: MvccSnapshot[applied={T|T < 7334451092142313472}]
I20260928 23:17:53.670842 11096 mvcc.cc:205] Tried to move back new op lower bound from 7334451092142313472 to 7334451091787624448. Current Snapshot: MvccSnapshot[applied={T|T < 7334451092142313472}]
I20260928 23:17:53.673414 10607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:53.673501 10607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:17:53.674166 10607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16ee68c61f594510a86eb669b8050d29: Sending DeleteTablet for 3 replicas of tablet 4930ae4ca4094c6da163f73ac7a3f25c
I20260928 23:17:53.674455 10867 tablet_service.cc:1514] Processing DeleteTablet for tablet 4930ae4ca4094c6da163f73ac7a3f25c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40164
I20260928 23:17:53.674476 11001 tablet_service.cc:1514] Processing DeleteTablet for tablet 4930ae4ca4094c6da163f73ac7a3f25c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36778
I20260928 23:17:53.674479 10733 tablet_service.cc:1514] Processing DeleteTablet for tablet 4930ae4ca4094c6da163f73ac7a3f25c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:17:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40352
I20260928 23:17:53.674713 11105 tablet_replica.cc:330] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: stopping tablet replica
I20260928 23:17:53.674804 11105 raft_consensus.cc:2229] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:17:53.674912 11105 raft_consensus.cc:2258] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:53.674993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 10670
I20260928 23:17:53.675050 11104 tablet_replica.cc:330] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: stopping tablet replica
I20260928 23:17:53.675128 11104 raft_consensus.cc:2229] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:17:53.675220 11104 raft_consensus.cc:2258] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:17:53.675987 11104 ts_tablet_manager.cc:1905] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:53.676026 11105 ts_tablet_manager.cc:1905] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:17:53.677821 11105 ts_tablet_manager.cc:1918] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:53.677894 11104 ts_tablet_manager.cc:1918] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:17:53.677961 11104 log.cc:1196] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/4930ae4ca4094c6da163f73ac7a3f25c
I20260928 23:17:53.678037 11105 log.cc:1196] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/4930ae4ca4094c6da163f73ac7a3f25c
I20260928 23:17:53.678200 11104 ts_tablet_manager.cc:1939] T 4930ae4ca4094c6da163f73ac7a3f25c P a9cf65412beb4870816ee04da7a130a2: Deleting consensus metadata
I20260928 23:17:53.678411 11105 ts_tablet_manager.cc:1939] T 4930ae4ca4094c6da163f73ac7a3f25c P 8b5b00a8c8974a838ea32cd645faf629: Deleting consensus metadata
I20260928 23:17:53.678675 10592 catalog_manager.cc:4841] TS a9cf65412beb4870816ee04da7a130a2 (127.0.98.2:33963): tablet 4930ae4ca4094c6da163f73ac7a3f25c (table test-table [id=092632856a9d4873ba98be712ae488d6]) successfully deleted
I20260928 23:17:53.678802 10595 catalog_manager.cc:4841] TS 8b5b00a8c8974a838ea32cd645faf629 (127.0.98.3:34073): tablet 4930ae4ca4094c6da163f73ac7a3f25c (table test-table [id=092632856a9d4873ba98be712ae488d6]) successfully deleted
W20260928 23:17:53.681318 10595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42365 (error 108)
I20260928 23:17:53.681488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 10804
I20260928 23:17:53.682334 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:17:53.682197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42365 (local address 127.0.0.1:40364)
0928 23:17:53.682254 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 23:17:53.682298 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 23:17:53.682495 10595 catalog_manager.cc:4584] TS 5ab85aa60faa438aaa4d8bd6ce398e7f (127.0.98.1:42365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4930ae4ca4094c6da163f73ac7a3f25c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42365: connect: Connection refused (error 111)
I20260928 23:17:53.686575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 10938
I20260928 23:17:53.690246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 10576
2026-09-28T23:17:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2163 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:17:53 dist-test-slave-djkr krb5kdc[11110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:17:53 dist-test-slave-djkr krb5kdc[11110](info): set up 2 sockets
Sep 28 23:17:53 dist-test-slave-djkr krb5kdc[11110](info): commencing operation
krb5kdc: starting...
W20260928 23:17:54.636163 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:17:54 dist-test-slave-djkr krb5kdc[11110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:17:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:17:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 23:17:54.778223 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNOOpNT/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/postgres -l logfile start
2026-09-28 23:17:57.044 UTC [11140] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 23:17:57.044 UTC [11140] LOG: listening on IPv4 address "127.0.98.20", port 44473
2026-09-28 23:17:57.047 UTC [11140] LOG: listening on Unix socket "/tmp/.s.PGSQL.44473"
2026-09-28 23:17:57.066 UTC [11143] LOG: database system was shut down at 2026-09-28 23:17:55 UTC
2026-09-28 23:17:57.070 UTC [11140] LOG: database system is ready to accept connections
I20260928 23:17:57.935979 392 mini_postgres.cc:96] Postgres bound to 44473
2026-09-28 23:17:57.941 UTC [11151] FATAL: database "slave" does not exist
127.0.98.20:44473 - accepting connections
I20260928 23:17:57.941932 392 mini_ranger.cc:161] Starting Ranger...
I20260928 23:17:57.955281 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 23:17:58.380465 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 23:17:58.380529 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-admin
2026-09-28 23:17:58,645 [I] DB FLAVOR :POSTGRES
2026-09-28 23:17:58,646 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:17:58,646 [I] Checking connection..
2026-09-28 23:17:58,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 23:17:58,878 [I] Checking connection passed.
2026-09-28 23:17:58,878 [I] --------- Verifying version history table ---------
2026-09-28 23:17:58,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:17:59,112 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:17:59,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:17:59,347 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:17:59,347 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 23:17:59,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 23:17:59.591 UTC [11215] WARNING: there is no transaction in progress
2026-09-28 23:17:59,596 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 23:17:59,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:17:59,845 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 23:17:59,845 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:17:59,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 23:18:00,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 23:18:00,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-djkr.c.gcp-upstream.internal','N') ;"
2026-09-28 23:18:00,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:18:00,810 [I] Table x_portal_user does not exist in database ranger
2026-09-28 23:18:00,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:18:01,029 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 23:18:01,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:18:01,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:18:01,504 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 23:18:01,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 23:18:02.406 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.420 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.435 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.449 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.464 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.478 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.528 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.535 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.549 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.558 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.568 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.579 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.588 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.594 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.601 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:02.607 UTC [11323] WARNING: there is no transaction in progress
2026-09-28 23:18:03,166 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 23:18:03,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 23:18:03,402 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 23:18:03,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:18:03,648 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 23:18:03,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:18:03,883 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 23:18:03,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:18:04,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:18:04,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-djkr.c.gcp-upstream.internal';"
2026-09-28 23:18:04,585 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 23:18:04,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 23:18:04,800 [I] DB_PATCHES have already been applied
I20260928 23:18:04.805737 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNOOpNT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 23:18:04.805794 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 23:18:04.808472 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:50135
Sep 28, 2026 11:18:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:18:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:18:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:18:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:18:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:18:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:18:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 23:18:05.785673 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.844s user 0.003s sys 0.004s
Sep 28, 2026 11:18:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:18:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 11:18:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:18:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:18:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 23:18:25.370718 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 23:18:25 dist-test-slave-djkr krb5kdc[11110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637505, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 23:18:25.455785 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 23:18:25.465018 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 23:18:26.201857 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 23:18:26.201920 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-kms
I20260928 23:18:26.201931 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:44473
2026-09-28 23:18:26,354 [I] DB FLAVOR :POSTGRES
2026-09-28 23:18:26,355 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:18:26,355 [I] Checking connection
2026-09-28 23:18:26,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 23:18:26,578 [I] connection success
2026-09-28 23:18:26,579 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 23:18:26,579 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 23:18:26,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 23:18:26,818 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 23:18:26,818 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:18:26,818 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 23:18:26,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 23:18:27,055 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 23:18:27.487627 392 mini_ranger_kms.cc:310] Created kms service
I20260928 23:18:27.657605 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 23:18:27.696888 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 23:18:27 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637505, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:18:28.071585 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 23:18:28.071848 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 23:18:28.071878 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 23:18:28.074724 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11503
I20260928 23:18:28.074839 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56239
Sep 28, 2026 11:18:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:18:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:18:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:18:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:18:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:18:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 23:18:30.001683 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.546s user 0.010s sys 0.126s
I20260928 23:18:30.001773 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 23:18:30.001852 392 mini_ranger_kms.cc:401] 127.0.98.20:56239/kms/v1/keys
Sep 28 23:18:30 dist-test-slave-djkr krb5kdc[11110](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790637510, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 23:18:30 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637510, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 23:18:30 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637510, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 23:18:33 dist-test-slave-djkr krb5kdc[11110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:18:33.122355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42131
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42131
--ranger_config_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56239/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:18:33.201447 11554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:18:33.201648 11554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:18:33.201678 11554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:18:33.202802 11554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:18:33.202842 11554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:18:33.202862 11554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:18:33.202875 11554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:18:33.202891 11554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:18:33.204421 11554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56239/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42131
--ranger_config_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:18:33.204643 11554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:18:33.204835 11554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:18:33.209478 11562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:18:33.209528 11559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:18:33.209584 11554 server_base.cc:1034] running on GCE node
W20260928 23:18:33.209690 11560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:18:33.210006 11554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:18:33.210281 11554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:18:33.211473 11554 hybrid_clock.cc:648] HybridClock initialized: now 1790637513211435 us; error 34 us; skew 500 ppm
Sep 28 23:18:33 dist-test-slave-djkr krb5kdc[11110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:18:33.213989 11554 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:18:33.214424 11554 webserver.cc:460] Webserver started at http://127.0.98.62:45721/ using document root <none> and password file <none>
I20260928 23:18:33.214617 11554 fs_manager.cc:359] Metadata directory not provided
I20260928 23:18:33.214654 11554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:18:33.214771 11554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:18:33 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:18:33.268385 11554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "92fd2e238f8345c38e17c1cc9c92ee9e"
format_stamp: "Formatted at 2026-09-28 23:18:33 on dist-test-slave-djkr"
server_key: "8f54ed3341bc7aec6d620bc0cf883884"
server_key_iv: "c80346e19cf51395125b7b30bfe6eec9"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:33.268877 11554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "92fd2e238f8345c38e17c1cc9c92ee9e"
format_stamp: "Formatted at 2026-09-28 23:18:33 on dist-test-slave-djkr"
server_key: "8f54ed3341bc7aec6d620bc0cf883884"
server_key_iv: "c80346e19cf51395125b7b30bfe6eec9"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:33.270613 11554 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 23:18:33.284384 11569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:18:33.284607 11554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:18:33.284711 11554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "92fd2e238f8345c38e17c1cc9c92ee9e"
format_stamp: "Formatted at 2026-09-28 23:18:33 on dist-test-slave-djkr"
server_key: "8f54ed3341bc7aec6d620bc0cf883884"
server_key_iv: "c80346e19cf51395125b7b30bfe6eec9"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:33.284806 11554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:18:33.304796 11554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:18:33.306635 11554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:18:33.306778 11554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:18:33.311483 11554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42131
I20260928 23:18:33.311506 11621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42131 every 8 connection(s)
I20260928 23:18:33.311878 11554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:18:33.312651 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:18:33.315737 11622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e: Bootstrap starting.
I20260928 23:18:33.316434 11622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e: Neither blocks nor log segments found. Creating new log.
I20260928 23:18:33.317225 11622 log.cc:824] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:18:33.317998 11622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e: No bootstrap required, opened a new log
I20260928 23:18:33.318537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 11554
I20260928 23:18:33.318617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:18:33.318778 392 external_mini_cluster.cc:1428] Setting key a57ec7196b9650c6474821eae5a212ae
I20260928 23:18:33.320827 11622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fd2e238f8345c38e17c1cc9c92ee9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } }
I20260928 23:18:33.321075 11622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:18:33.321133 11622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fd2e238f8345c38e17c1cc9c92ee9e, State: Initialized, Role: FOLLOWER
I20260928 23:18:33.321254 11622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fd2e238f8345c38e17c1cc9c92ee9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } }
I20260928 23:18:33.321332 11622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:18:33.321394 11622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:18:33.321444 11622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:18:33.322093 11622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fd2e238f8345c38e17c1cc9c92ee9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } }
I20260928 23:18:33.322201 11622 leader_election.cc:302] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92fd2e238f8345c38e17c1cc9c92ee9e; no voters:
I20260928 23:18:33.322403 11622 leader_election.cc:288] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:18:33.322438 11627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:18:33.322682 11627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [term 1 LEADER]: Becoming Leader. State: Replica: 92fd2e238f8345c38e17c1cc9c92ee9e, State: Running, Role: LEADER
I20260928 23:18:33.322782 11622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:18:33.322798 11627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fd2e238f8345c38e17c1cc9c92ee9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } }
I20260928 23:18:33.323240 11627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92fd2e238f8345c38e17c1cc9c92ee9e. Latest consensus state: current_term: 1 leader_uuid: "92fd2e238f8345c38e17c1cc9c92ee9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fd2e238f8345c38e17c1cc9c92ee9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } }
I20260928 23:18:33.323244 11629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92fd2e238f8345c38e17c1cc9c92ee9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fd2e238f8345c38e17c1cc9c92ee9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42131 } } }
I20260928 23:18:33.323315 11627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [sys.catalog]: This master's current role is: LEADER
I20260928 23:18:33.323338 11629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e [sys.catalog]: This master's current role is: LEADER
I20260928 23:18:33.325079 11622 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 28 23:18:33 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:18:33.331377 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:33.319262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.0.98.62:42131)
0928 23:18:33.319443 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:33.319452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:33.319486 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:18:33.320789 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:33.320796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:33.320851 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:33.320983 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:18:33.321706 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:33.322873 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:33.323917 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:33.324542 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:33.324651 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:33.327468 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:18:33.327499 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:18:33.327500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:18:33.327568 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:18:33.329706 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:18:33.330093 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:18:33.330097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:18:33.330097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:18:33.330192 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:18:33.330398 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:18:33.330399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:18:33.330399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:18:33.330719 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:18:33.330867 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:33.331000 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 23:18:33.331151 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260928 23:18:33.331691 11631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:18:33.332181 11631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:18:33.334613 11631 catalog_manager.cc:1347] Generated new cluster ID: 7bafe7c6cc8c4c7c9ec67b8b72f9ebcb
I20260928 23:18:33.334672 11631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:18:33.359932 11631 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:18:33.360778 11631 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:18:33.372335 11631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e: Generated new TSK 0
I20260928 23:18:33.372632 11631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:18:34.119585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42131
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:18:34.219764 11667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:18:34.220037 11667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:18:34.220072 11667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:18:34.221707 11667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:18:34.221760 11667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:18:34.221796 11667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:18:34.223644 11667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:18:34.224012 11667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:18:34.224371 11667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:18:34.230020 11672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:18:34.230053 11676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:18:34.230365 11674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:18:34.231798 11667 server_base.cc:1034] running on GCE node
I20260928 23:18:34.232108 11667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:18:34.232375 11667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:18:34.233551 11667 hybrid_clock.cc:648] HybridClock initialized: now 1790637514233529 us; error 35 us; skew 500 ppm
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:18:34.236187 11667 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:18:34.236627 11667 webserver.cc:460] Webserver started at http://127.0.98.1:36447/ using document root <none> and password file <none>
I20260928 23:18:34.236833 11667 fs_manager.cc:359] Metadata directory not provided
I20260928 23:18:34.236881 11667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:18:34.237006 11667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:18:34.255148 11667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "2382ca5a7ca04a0cacd0eed48546d436"
format_stamp: "Formatted at 2026-09-28 23:18:34 on dist-test-slave-djkr"
server_key: "4ccbd96793c3066508e1119d89f19b41"
server_key_iv: "28792172d901b90d98a68fd9b3dd50e1"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:34.255606 11667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2382ca5a7ca04a0cacd0eed48546d436"
format_stamp: "Formatted at 2026-09-28 23:18:34 on dist-test-slave-djkr"
server_key: "4ccbd96793c3066508e1119d89f19b41"
server_key_iv: "28792172d901b90d98a68fd9b3dd50e1"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:34.257330 11667 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 23:18:34.280172 11684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:18:34.280372 11667 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260928 23:18:34.280479 11667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "2382ca5a7ca04a0cacd0eed48546d436"
format_stamp: "Formatted at 2026-09-28 23:18:34 on dist-test-slave-djkr"
server_key: "4ccbd96793c3066508e1119d89f19b41"
server_key_iv: "28792172d901b90d98a68fd9b3dd50e1"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:34.280602 11667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:18:34.306669 11667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:18:34.309003 11667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:18:34.309134 11667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:18:34.309378 11667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:18:34.309751 11667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:18:34.309801 11667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:18:34.309831 11667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:18:34.309870 11667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:18:34.317001 11667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33563
I20260928 23:18:34.317346 11667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:18:34.323740 11797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33563 every 8 connection(s)
I20260928 23:18:34.327142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 11667
I20260928 23:18:34.327231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:18:34.327406 392 external_mini_cluster.cc:1428] Setting key 66e1f34db9e92c4f22cb3bb7a3dbb16b
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:18:34.335988 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.324449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42131 (local address 127.0.98.1:48231)
0928 23:18:34.325217 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 23:18:34.325267 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:18:34.325885 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:18:34.326301 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:18:34.326307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:18:34.326545 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:18:34.326833 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.326840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.328502 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.328503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.329813 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.329818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.330281 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.330282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.330419 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.330971 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:18:34.330981 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:18:34.332499 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:18:34.334378 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:18:34.334384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:18:34.334386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:18:34.334671 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:18:34.334899 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:18:34.334900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:18:34.334900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:18:34.335035 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:18:34.335328 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:18:34.335335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:18:34.335506 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 23:18:34.335624 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 23:18:34.335768 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":84,"threads_started":1}
I20260928 23:18:34.336328 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.324456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48231 (local address 127.0.98.62:42131)
0928 23:18:34.326083 (+ 1627us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:34.326085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:34.326094 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:18:34.326118 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:34.326119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:34.326131 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:34.326925 (+ 794us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:18:34.326965 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.328395 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.329913 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.330184 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.330666 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.332608 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:18:34.332609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:18:34.332610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:18:34.332617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:18:34.334286 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:18:34.334757 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:18:34.334758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:18:34.334759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:18:34.334818 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:18:34.335104 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:18:34.335105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:18:34.335105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:18:34.335232 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:18:34.336108 (+ 876us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:34.336149 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 23:18:34.336229 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1597,"thread_start_us":655,"threads_started":1}
I20260928 23:18:34.336740 11798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42131
I20260928 23:18:34.336823 11798 heartbeater.cc:464] Registering TS with master...
I20260928 23:18:34.336987 11798 heartbeater.cc:511] Master 127.0.98.62:42131 requested a full tablet report, sending...
I20260928 23:18:34.337507 11586 ts_manager.cc:194] Registered new tserver with Master: 2382ca5a7ca04a0cacd0eed48546d436 (127.0.98.1:33563)
I20260928 23:18:34.338397 11586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48231
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:18:34.357668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42131
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 23:18:34.477281 11806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:18:34.477538 11806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:18:34.477568 11806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:18:34.479187 11806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:18:34.479235 11806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:18:34.479270 11806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:18:34.481333 11806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:18:34.481554 11806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:18:34.481782 11806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:18:34.487207 11811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:18:34.487247 11812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:18:34.487615 11814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:18:34.487709 11806 server_base.cc:1034] running on GCE node
I20260928 23:18:34.488005 11806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:18:34.488299 11806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:18:34.495661 11806 hybrid_clock.cc:648] HybridClock initialized: now 1790637514494402 us; error 1270 us; skew 500 ppm
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:18:34.498806 11806 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:18:34.499338 11806 webserver.cc:460] Webserver started at http://127.0.98.2:36411/ using document root <none> and password file <none>
I20260928 23:18:34.499617 11806 fs_manager.cc:359] Metadata directory not provided
I20260928 23:18:34.499678 11806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:18:34.499852 11806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:18:34.513897 11806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "e1125af52c034353b0f7009416e5e24d"
format_stamp: "Formatted at 2026-09-28 23:18:34 on dist-test-slave-djkr"
server_key: "f95a604e08bf3b07b7f2899e8c779fc1"
server_key_iv: "c7b83a253f263740b8d851b2f486c2c7"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:34.514640 11806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e1125af52c034353b0f7009416e5e24d"
format_stamp: "Formatted at 2026-09-28 23:18:34 on dist-test-slave-djkr"
server_key: "f95a604e08bf3b07b7f2899e8c779fc1"
server_key_iv: "c7b83a253f263740b8d851b2f486c2c7"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:34.524508 11806 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.000s
I20260928 23:18:34.538592 11822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:18:34.539301 11806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:18:34.539404 11806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "e1125af52c034353b0f7009416e5e24d"
format_stamp: "Formatted at 2026-09-28 23:18:34 on dist-test-slave-djkr"
server_key: "f95a604e08bf3b07b7f2899e8c779fc1"
server_key_iv: "c7b83a253f263740b8d851b2f486c2c7"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:34.539506 11806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:18:34.554020 11806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:18:34.556434 11806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:18:34.556627 11806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:18:34.556860 11806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:18:34.557407 11806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:18:34.557503 11806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:18:34.557578 11806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:18:34.557636 11806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:18:34.564236 11806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36475
I20260928 23:18:34.564507 11935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36475 every 8 connection(s)
I20260928 23:18:34.564692 11806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:18:34.567523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 11806
I20260928 23:18:34.567633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:18:34.567816 392 external_mini_cluster.cc:1428] Setting key d3704a642295112d9dd8a3b4a65db5eb
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:18:34.577739 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.565663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50409 (local address 127.0.98.62:42131)
0928 23:18:34.565706 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:34.565709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:34.565896 (+ 187us) server_negotiation.cc:407] Connection header received
0928 23:18:34.567106 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:34.567107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:34.567118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:34.567147 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:18:34.568017 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.568777 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.570281 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.570593 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.570638 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.574481 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:18:34.574482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:18:34.574483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:18:34.574490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:18:34.576163 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:18:34.576626 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:18:34.576627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:18:34.576628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:18:34.576690 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:18:34.576968 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:18:34.576969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:18:34.576969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:18:34.577116 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:18:34.577157 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:34.577551 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 23:18:34.577641 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:18:34.578508 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.565397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42131 (local address 127.0.98.2:50409)
0928 23:18:34.565776 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 23:18:34.565818 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:18:34.566958 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:18:34.567328 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:18:34.567332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:18:34.567578 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:18:34.567897 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.567904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.568887 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.568888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.570183 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.570186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.570725 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.570726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.570859 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.571398 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:18:34.571408 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:18:34.572981 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:18:34.576255 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:18:34.576261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:18:34.576263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:18:34.576539 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:18:34.576758 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:18:34.576758 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:18:34.576759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:18:34.576893 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:18:34.577205 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:18:34.577212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:18:34.577390 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 23:18:34.578116 (+ 726us) client_negotiation.cc:240] Negotiation successful
0928 23:18:34.578279 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":75,"threads_started":1}
I20260928 23:18:34.580565 11936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42131
I20260928 23:18:34.580659 11936 heartbeater.cc:464] Registering TS with master...
I20260928 23:18:34.580847 11936 heartbeater.cc:511] Master 127.0.98.62:42131 requested a full tablet report, sending...
I20260928 23:18:34.581297 11586 ts_manager.cc:194] Registered new tserver with Master: e1125af52c034353b0f7009416e5e24d (127.0.98.2:36475)
I20260928 23:18:34.581766 11586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50409
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:18:34.618690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56239/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42131
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:18:34.751194 11941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:18:34.751523 11941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:18:34.751636 11941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:18:34.753293 11941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:18:34.753388 11941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:18:34.753465 11941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:18:34.755761 11941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56239/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:18:34.756192 11941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:18:34.756505 11941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:18:34.764418 11949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:18:34.764590 11947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:18:34.764974 11946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:18:34.766057 11941 server_base.cc:1034] running on GCE node
I20260928 23:18:34.766367 11941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:18:34.766654 11941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:18:34.768049 11941 hybrid_clock.cc:648] HybridClock initialized: now 1790637514768024 us; error 37 us; skew 500 ppm
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:18:34.771131 11941 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:18:34.771673 11941 webserver.cc:460] Webserver started at http://127.0.98.3:45021/ using document root <none> and password file <none>
I20260928 23:18:34.771901 11941 fs_manager.cc:359] Metadata directory not provided
I20260928 23:18:34.771955 11941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:18:34.772078 11941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:18:34.785634 11941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "7a2bbf5bb34f4298a6a06cd66a609f26"
format_stamp: "Formatted at 2026-09-28 23:18:34 on dist-test-slave-djkr"
server_key: "da18775c6f023dbb5e59de69850e752d"
server_key_iv: "48214c6432eb955f7e042c68ef4332d1"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:34.786155 11941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7a2bbf5bb34f4298a6a06cd66a609f26"
format_stamp: "Formatted at 2026-09-28 23:18:34 on dist-test-slave-djkr"
server_key: "da18775c6f023dbb5e59de69850e752d"
server_key_iv: "48214c6432eb955f7e042c68ef4332d1"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:34.787964 11941 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:18:34.815641 11956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:18:34.815904 11941 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260928 23:18:34.816000 11941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "7a2bbf5bb34f4298a6a06cd66a609f26"
format_stamp: "Formatted at 2026-09-28 23:18:34 on dist-test-slave-djkr"
server_key: "da18775c6f023dbb5e59de69850e752d"
server_key_iv: "48214c6432eb955f7e042c68ef4332d1"
server_key_version: "kuduclusterkey@0"
I20260928 23:18:34.816100 11941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:18:34.829869 11941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:18:34.834259 11941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:18:34.834460 11941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:18:34.835120 11941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:18:34.836376 11941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:18:34.837085 11941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260928 23:18:34.837184 11941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:18:34.837248 11941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:18:34.848203 11941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42791
I20260928 23:18:34.848701 11941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:18:34.849915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 11941
I20260928 23:18:34.850013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:18:34.850204 392 external_mini_cluster.cc:1428] Setting key f0325d76452817917473f443af245f07
I20260928 23:18:34.856643 12069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42791 every 8 connection(s)
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:18:34.878731 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.858397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33051 (local address 127.0.98.62:42131)
0928 23:18:34.858440 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:34.858442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:34.866787 (+ 8345us) server_negotiation.cc:407] Connection header received
0928 23:18:34.868285 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:34.868286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:34.868298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:34.868328 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:18:34.869218 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.870007 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.872107 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.872372 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.872419 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.875656 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:18:34.875657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:18:34.875657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:18:34.875664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:18:34.877244 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:18:34.877721 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:18:34.877723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:18:34.877723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:18:34.877779 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:18:34.878020 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:18:34.878021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:18:34.878021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:18:34.878149 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:18:34.878197 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:34.878562 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 23:18:34.878639 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:18:34.879446 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.858137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42131 (local address 127.0.98.3:33051)
0928 23:18:34.866626 (+ 8489us) negotiation.cc:107] Waiting for socket to connect
0928 23:18:34.866685 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 23:18:34.868157 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:18:34.868537 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:18:34.868542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:18:34.868808 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:18:34.869101 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.869109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.870121 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.870121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.871394 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.871396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.872526 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.872527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.872670 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.873855 (+ 1185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:18:34.873865 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:18:34.875525 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:18:34.877337 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:18:34.877343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:18:34.877346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:18:34.877636 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:18:34.877837 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:18:34.877838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:18:34.877838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:18:34.877953 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:18:34.878229 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:18:34.878234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:18:34.878399 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 23:18:34.879066 (+ 667us) client_negotiation.cc:240] Negotiation successful
0928 23:18:34.879222 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8436,"thread_start_us":76,"threads_started":1}
I20260928 23:18:34.880081 12070 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42131
I20260928 23:18:34.880172 12070 heartbeater.cc:464] Registering TS with master...
I20260928 23:18:34.880353 12070 heartbeater.cc:511] Master 127.0.98.62:42131 requested a full tablet report, sending...
I20260928 23:18:34.880797 11586 ts_manager.cc:194] Registered new tserver with Master: 7a2bbf5bb34f4298a6a06cd66a609f26 (127.0.98.3:42791)
I20260928 23:18:34.881269 11586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33051
I20260928 23:18:34.883934 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:18:34.897624 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.885743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.98.62:42131)
0928 23:18:34.885788 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:34.885790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:34.886113 (+ 323us) server_negotiation.cc:407] Connection header received
0928 23:18:34.886202 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:34.886203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:34.886212 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:34.886238 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:18:34.890503 (+ 4265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.891213 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.892878 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.893151 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.893194 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.894671 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:18:34.894673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:18:34.894673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:18:34.894680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:18:34.896236 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:18:34.896807 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:18:34.896808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:18:34.896808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:18:34.896865 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:18:34.897061 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:18:34.897062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:18:34.897063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:18:34.897179 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:18:34.897249 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:34.897459 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 23:18:34.897545 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:18:34.899534 11586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:18:34.901693 11586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:18:34.923079 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.908800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36475 (local address 127.0.0.1:35552)
0928 23:18:34.910172 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0928 23:18:34.910212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:18:34.910267 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:18:34.911349 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:18:34.911349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:18:34.911350 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:18:34.911375 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.911376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.913341 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.913342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.921712 (+ 8370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.921715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.922755 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.922756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.922840 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.922854 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:18:34.922924 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:18:34.922984 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"thread_start_us":64,"threads_started":1}
I20260928 23:18:34.923772 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.909643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33563 (local address 127.0.0.1:49066)
0928 23:18:34.911979 (+ 2336us) negotiation.cc:107] Waiting for socket to connect
0928 23:18:34.911988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:18:34.912030 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:18:34.913738 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:18:34.913739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:18:34.913742 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:18:34.913781 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.913782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.916937 (+ 3155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.916938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.918047 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.918049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.923469 (+ 5420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.923470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.923594 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.923600 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:18:34.923656 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 23:18:34.923695 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2322,"thread_start_us":71,"threads_started":1}
I20260928 23:18:34.925643 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.908997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42791 (local address 127.0.0.1:42904)
0928 23:18:34.910700 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0928 23:18:34.910709 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:18:34.910750 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:18:34.911221 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:18:34.911224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:18:34.911226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:18:34.911289 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.911291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.918964 (+ 7673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.918965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.920330 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:34.920332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.925418 (+ 5086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.925418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:34.925491 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.925495 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:18:34.925537 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:18:34.925575 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1687,"thread_start_us":51,"threads_started":1}
I20260928 23:18:34.926177 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.909826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.0.98.1:33563)
0928 23:18:34.910840 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:34.910844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:34.912081 (+ 1237us) server_negotiation.cc:407] Connection header received
0928 23:18:34.912117 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:34.912119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:34.912135 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:34.912191 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:18:34.913861 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.915154 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.920411 (+ 5257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.921248 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.921318 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.921386 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:34.926051 (+ 4665us) server_negotiation.cc:299] Negotiation successful
0928 23:18:34.926104 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":41,"threads_started":1}
I20260928 23:18:34.926515 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.909252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.0.98.2:36475)
0928 23:18:34.910392 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:34.910397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:34.910405 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:18:34.910441 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:34.910443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:34.910457 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:34.910529 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:18:34.911434 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.913249 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.921824 (+ 8575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.922571 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.922643 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.922720 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:34.926379 (+ 3659us) server_negotiation.cc:299] Negotiation successful
0928 23:18:34.926425 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":65,"threads_started":1}
I20260928 23:18:34.927311 11725 tablet_service.cc:1467] Processing CreateTablet for tablet feadc55713194179b34d9354a304bfee (DEFAULT_TABLE table=test-table [id=afa4da60a54f4cb4bb16ac54687c79db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:18:34.927788 11725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feadc55713194179b34d9354a304bfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:18:34.931665 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:34.909392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.0.98.3:42791)
0928 23:18:34.910652 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:34.910656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:34.910860 (+ 204us) server_negotiation.cc:407] Connection header received
0928 23:18:34.910896 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:34.910898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:34.910912 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:34.911136 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:18:34.911588 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.918872 (+ 7284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.921443 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:34.925247 (+ 3804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:34.925315 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:34.931490 (+ 6175us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:34.931521 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 23:18:34.931586 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1225,"thread_start_us":70,"threads_started":1}
I20260928 23:18:34.932430 11857 tablet_service.cc:1467] Processing CreateTablet for tablet feadc55713194179b34d9354a304bfee (DEFAULT_TABLE table=test-table [id=afa4da60a54f4cb4bb16ac54687c79db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:18:34.932751 11857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feadc55713194179b34d9354a304bfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:18:34.934971 12089 tablet_bootstrap.cc:492] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436: Bootstrap starting.
I20260928 23:18:34.935654 12089 tablet_bootstrap.cc:654] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436: Neither blocks nor log segments found. Creating new log.
I20260928 23:18:34.935910 12089 log.cc:824] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436: Log is configured to *not* fsync() on all Append() calls
I20260928 23:18:34.936146 12004 tablet_service.cc:1467] Processing CreateTablet for tablet feadc55713194179b34d9354a304bfee (DEFAULT_TABLE table=test-table [id=afa4da60a54f4cb4bb16ac54687c79db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:18:34.936434 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feadc55713194179b34d9354a304bfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:18:34.940447 12091 tablet_bootstrap.cc:492] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: Bootstrap starting.
I20260928 23:18:34.941406 12091 tablet_bootstrap.cc:654] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: Neither blocks nor log segments found. Creating new log.
I20260928 23:18:34.941746 12091 log.cc:824] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: Log is configured to *not* fsync() on all Append() calls
I20260928 23:18:34.942102 12092 tablet_bootstrap.cc:492] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: Bootstrap starting.
I20260928 23:18:34.943035 12092 tablet_bootstrap.cc:654] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: Neither blocks nor log segments found. Creating new log.
I20260928 23:18:34.943347 12092 log.cc:824] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:18:34.944197 12089 tablet_bootstrap.cc:492] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436: No bootstrap required, opened a new log
I20260928 23:18:34.944311 12089 ts_tablet_manager.cc:1397] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
Sep 28 23:18:34 dist-test-slave-djkr krb5kdc[11110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:18:34.947893 12091 tablet_bootstrap.cc:492] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: No bootstrap required, opened a new log
I20260928 23:18:34.948012 12091 ts_tablet_manager.cc:1397] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260928 23:18:34.949872 12091 raft_consensus.cc:348] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 } } peers { permanent_uuid: "e1125af52c034353b0f7009416e5e24d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 } }
I20260928 23:18:34.947387 12089 raft_consensus.cc:348] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 } } peers { permanent_uuid: "e1125af52c034353b0f7009416e5e24d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 } }
I20260928 23:18:34.950237 12089 raft_consensus.cc:374] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:18:34.950314 12089 raft_consensus.cc:729] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2382ca5a7ca04a0cacd0eed48546d436, State: Initialized, Role: FOLLOWER
I20260928 23:18:34.950518 12089 consensus_queue.cc:260] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 } } peers { permanent_uuid: "e1125af52c034353b0f7009416e5e24d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 } }
I20260928 23:18:34.950861 12089 ts_tablet_manager.cc:1428] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260928 23:18:34.951009 11798 heartbeater.cc:503] Master 127.0.98.62:42131 was elected leader, sending a full tablet report...
I20260928 23:18:34.952301 12092 tablet_bootstrap.cc:492] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: No bootstrap required, opened a new log
I20260928 23:18:34.952415 12092 ts_tablet_manager.cc:1397] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 23:18:34.952755 12091 raft_consensus.cc:374] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:18:34.952852 12091 raft_consensus.cc:729] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2bbf5bb34f4298a6a06cd66a609f26, State: Initialized, Role: FOLLOWER
I20260928 23:18:34.953029 12091 consensus_queue.cc:260] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 } } peers { permanent_uuid: "e1125af52c034353b0f7009416e5e24d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 } }
I20260928 23:18:34.954816 12092 raft_consensus.cc:348] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 } } peers { permanent_uuid: "e1125af52c034353b0f7009416e5e24d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 } }
I20260928 23:18:34.955216 12091 ts_tablet_manager.cc:1428] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260928 23:18:34.955359 12070 heartbeater.cc:503] Master 127.0.98.62:42131 was elected leader, sending a full tablet report...
I20260928 23:18:34.956033 12092 raft_consensus.cc:374] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:18:34.956135 12092 raft_consensus.cc:729] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1125af52c034353b0f7009416e5e24d, State: Initialized, Role: FOLLOWER
I20260928 23:18:34.956305 12092 consensus_queue.cc:260] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 } } peers { permanent_uuid: "e1125af52c034353b0f7009416e5e24d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 } }
I20260928 23:18:34.956650 12092 ts_tablet_manager.cc:1428] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:18:34.956784 11936 heartbeater.cc:503] Master 127.0.98.62:42131 was elected leader, sending a full tablet report...
I20260928 23:18:35.014602 12097 raft_consensus.cc:482] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:18:35.014748 12097 raft_consensus.cc:504] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 } } peers { permanent_uuid: "e1125af52c034353b0f7009416e5e24d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 } }
I20260928 23:18:35.015156 12097 leader_election.cc:288] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2382ca5a7ca04a0cacd0eed48546d436 (127.0.98.1:33563), 7a2bbf5bb34f4298a6a06cd66a609f26 (127.0.98.3:42791)
I20260928 23:18:35.019009 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:35.015445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33563 (local address 127.0.98.2:48739)
0928 23:18:35.015690 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 23:18:35.015703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:18:35.015755 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:18:35.015958 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:18:35.015959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:18:35.015962 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:18:35.016060 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:35.016062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:35.016943 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:35.016944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:35.018070 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:35.018072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:35.018785 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:35.018786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:35.018850 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:35.018853 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:18:35.018891 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 23:18:35.018926 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260928 23:18:35.019640 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:35.015698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48739 (local address 127.0.98.1:33563)
0928 23:18:35.015747 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:35.015749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:35.015762 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:18:35.015828 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:35.015828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:35.015837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:35.015904 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:18:35.016127 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:35.016812 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:35.018161 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:35.018690 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:35.018747 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:35.018757 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:35.019521 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 23:18:35.019579 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:18:35.019876 11752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feadc55713194179b34d9354a304bfee" candidate_uuid: "e1125af52c034353b0f7009416e5e24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2382ca5a7ca04a0cacd0eed48546d436" is_pre_election: true
I20260928 23:18:35.020020 11752 raft_consensus.cc:2454] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1125af52c034353b0f7009416e5e24d in term 0.
I20260928 23:18:35.020404 11823 leader_election.cc:302] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2382ca5a7ca04a0cacd0eed48546d436, e1125af52c034353b0f7009416e5e24d; no voters:
I20260928 23:18:35.020535 12097 raft_consensus.cc:2781] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:18:35.020578 12097 raft_consensus.cc:482] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:18:35.020597 12097 raft_consensus.cc:3037] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:18:35.021440 12097 raft_consensus.cc:504] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 } } peers { permanent_uuid: "e1125af52c034353b0f7009416e5e24d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 } }
I20260928 23:18:35.021569 12097 leader_election.cc:288] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [CANDIDATE]: Term 1 election: Requested vote from peers 2382ca5a7ca04a0cacd0eed48546d436 (127.0.98.1:33563), 7a2bbf5bb34f4298a6a06cd66a609f26 (127.0.98.3:42791)
I20260928 23:18:35.021728 11752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feadc55713194179b34d9354a304bfee" candidate_uuid: "e1125af52c034353b0f7009416e5e24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2382ca5a7ca04a0cacd0eed48546d436"
I20260928 23:18:35.021812 11752 raft_consensus.cc:3037] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:18:35.022678 11752 raft_consensus.cc:2454] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1125af52c034353b0f7009416e5e24d in term 1.
I20260928 23:18:35.022891 11823 leader_election.cc:302] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2382ca5a7ca04a0cacd0eed48546d436, e1125af52c034353b0f7009416e5e24d; no voters:
I20260928 23:18:35.022989 12097 raft_consensus.cc:2781] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:18:35.023126 12097 raft_consensus.cc:686] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 1 LEADER]: Becoming Leader. State: Replica: e1125af52c034353b0f7009416e5e24d, State: Running, Role: LEADER
I20260928 23:18:35.023219 12097 consensus_queue.cc:237] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 } } peers { permanent_uuid: "e1125af52c034353b0f7009416e5e24d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } } peers { permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 } }
I20260928 23:18:35.024147 11586 catalog_manager.cc:5495] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d reported cstate change: term changed from 0 to 1, leader changed from <none> to e1125af52c034353b0f7009416e5e24d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e1125af52c034353b0f7009416e5e24d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1125af52c034353b0f7009416e5e24d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 } health_report { overall_health: UNKNOWN } } }
I20260928 23:18:35.024637 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:35.015935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42791 (local address 127.0.98.2:50177)
0928 23:18:35.019078 (+ 3143us) negotiation.cc:107] Waiting for socket to connect
0928 23:18:35.019087 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:18:35.019119 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:18:35.021713 (+ 2594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:18:35.021715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:18:35.021716 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:18:35.021746 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:35.021747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:35.022606 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:35.022607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:35.023691 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:18:35.023693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:18:35.024418 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:18:35.024419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:18:35.024499 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:35.024503 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:18:35.024543 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 23:18:35.024574 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3134,"thread_start_us":111,"threads_started":1}
I20260928 23:18:35.025496 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:35.016146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50177 (local address 127.0.98.3:42791)
0928 23:18:35.016912 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:35.016914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:35.019164 (+ 2250us) server_negotiation.cc:407] Connection header received
0928 23:18:35.019190 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:35.019191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:35.019202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:35.019241 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:18:35.021827 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:35.022512 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:35.023771 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:35.024271 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:35.024334 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:35.024345 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:35.025392 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0928 23:18:35.025432 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739}
I20260928 23:18:35.025777 12022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feadc55713194179b34d9354a304bfee" candidate_uuid: "e1125af52c034353b0f7009416e5e24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" is_pre_election: true
I20260928 23:18:35.025915 12022 raft_consensus.cc:2454] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1125af52c034353b0f7009416e5e24d in term 0.
I20260928 23:18:35.026031 12024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feadc55713194179b34d9354a304bfee" candidate_uuid: "e1125af52c034353b0f7009416e5e24d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26"
I20260928 23:18:35.026081 12024 raft_consensus.cc:3037] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:18:35.026863 12024 raft_consensus.cc:2454] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1125af52c034353b0f7009416e5e24d in term 1.
I20260928 23:18:35.034636 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:35.031326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.0.98.2:36475)
0928 23:18:35.031373 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:18:35.031375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:18:35.031516 (+ 141us) server_negotiation.cc:407] Connection header received
0928 23:18:35.031568 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:18:35.031569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:18:35.031588 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:18:35.031643 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:18:35.031828 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:35.032501 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:35.033578 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:18:35.033879 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:18:35.033931 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:18:35.034114 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:18:35.034234 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:18:35.034299 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:18:35.034524 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 23:18:35.034544 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:18:35.036628 12024 raft_consensus.cc:1261] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26 [term 1 FOLLOWER]: Refusing update from remote peer e1125af52c034353b0f7009416e5e24d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:18:35.036972 12097 consensus_queue.cc:1035] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a2bbf5bb34f4298a6a06cd66a609f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:18:35.038931 11752 raft_consensus.cc:1261] T feadc55713194179b34d9354a304bfee P 2382ca5a7ca04a0cacd0eed48546d436 [term 1 FOLLOWER]: Refusing update from remote peer e1125af52c034353b0f7009416e5e24d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:18:35.039238 12097 consensus_queue.cc:1035] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2382ca5a7ca04a0cacd0eed48546d436" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:18:35.043454 12102 mvcc.cc:205] Tried to move back new op lower bound from 7334451261587296256 to 7334451261530468352. Current Snapshot: MvccSnapshot[applied={T|T < 7334451261587296256}]
I20260928 23:18:35.043668 12105 mvcc.cc:205] Tried to move back new op lower bound from 7334451261587296256 to 7334451261530468352. Current Snapshot: MvccSnapshot[applied={T|T < 7334451261587296256}]
I20260928 23:18:35.045408 12103 mvcc.cc:205] Tried to move back new op lower bound from 7334451261587296256 to 7334451261530468352. Current Snapshot: MvccSnapshot[applied={T|T < 7334451261587296256}]
I20260928 23:18:35.047950 11586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:18:35.048048 11586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:18:35.048754 11586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92fd2e238f8345c38e17c1cc9c92ee9e: Sending DeleteTablet for 3 replicas of tablet feadc55713194179b34d9354a304bfee
I20260928 23:18:35.049177 11857 tablet_service.cc:1514] Processing DeleteTablet for tablet feadc55713194179b34d9354a304bfee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:18:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35552
I20260928 23:18:35.049365 12004 tablet_service.cc:1514] Processing DeleteTablet for tablet feadc55713194179b34d9354a304bfee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:18:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42904
I20260928 23:18:35.049680 11725 tablet_service.cc:1514] Processing DeleteTablet for tablet feadc55713194179b34d9354a304bfee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:18:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49066
I20260928 23:18:35.050729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 11667
I20260928 23:18:35.055644 12112 tablet_replica.cc:330] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: stopping tablet replica
I20260928 23:18:35.055799 12112 raft_consensus.cc:2229] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:18:35.056000 12112 raft_consensus.cc:2258] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:18:35.059566 12113 tablet_replica.cc:330] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: stopping tablet replica
I20260928 23:18:35.059743 12113 raft_consensus.cc:2229] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:18:35.059789 12112 ts_tablet_manager.cc:1905] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:18:35.059928 12113 raft_consensus.cc:2258] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:18:35.061558 12112 ts_tablet_manager.cc:1918] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:18:35.061659 12112 log.cc:1196] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/feadc55713194179b34d9354a304bfee
W20260928 23:18:35.061882 11570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33563 (error 108)
I20260928 23:18:35.061913 12112 ts_tablet_manager.cc:1939] T feadc55713194179b34d9354a304bfee P e1125af52c034353b0f7009416e5e24d: Deleting consensus metadata
I20260928 23:18:35.062287 11571 catalog_manager.cc:4841] TS e1125af52c034353b0f7009416e5e24d (127.0.98.2:36475): tablet feadc55713194179b34d9354a304bfee (table test-table [id=afa4da60a54f4cb4bb16ac54687c79db]) successfully deleted
I20260928 23:18:35.062448 12113 ts_tablet_manager.cc:1905] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:18:35.064039 12113 ts_tablet_manager.cc:1918] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:18:35.064113 12113 log.cc:1196] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/feadc55713194179b34d9354a304bfee
I20260928 23:18:35.064651 12113 ts_tablet_manager.cc:1939] T feadc55713194179b34d9354a304bfee P 7a2bbf5bb34f4298a6a06cd66a609f26: Deleting consensus metadata
I20260928 23:18:35.065117 11573 catalog_manager.cc:4841] TS 7a2bbf5bb34f4298a6a06cd66a609f26 (127.0.98.3:42791): tablet feadc55713194179b34d9354a304bfee (table test-table [id=afa4da60a54f4cb4bb16ac54687c79db]) successfully deleted
I20260928 23:18:35.067696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 11806
I20260928 23:18:35.067992 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:18:35.067864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33563 (local address 127.0.0.1:49074)
0928 23:18:35.067920 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 23:18:35.067957 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 23:18:35.068655 11570 catalog_manager.cc:4584] TS 2382ca5a7ca04a0cacd0eed48546d436 (127.0.98.1:33563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet feadc55713194179b34d9354a304bfee: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33563: connect: Connection refused (error 111)
I20260928 23:18:35.073683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 11941
I20260928 23:18:35.079370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 11554
I20260928 23:18:35.084100 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 23:18:35.427692 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 23:18:35.427799 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 23:18:35.761200 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 23:18:35.761 UTC [11140] LOG: received smart shutdown request
2026-09-28 23:18:35.763 UTC [11140] LOG: background worker "logical replication launcher" (PID 11149) exited with exit code 1
2026-09-28 23:18:35.764 UTC [11144] LOG: shutting down
2026-09-28 23:18:35.905 UTC [11140] LOG: database system is shut down
2026-09-28T23:18:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42301 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:18:36 dist-test-slave-djkr krb5kdc[12124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:18:36 dist-test-slave-djkr krb5kdc[12124](info): set up 2 sockets
Sep 28 23:18:36 dist-test-slave-djkr krb5kdc[12124](info): commencing operation
krb5kdc: starting...
W20260928 23:18:36.939216 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:18:37 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:18:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:18:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 23:18:37.079157 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNOOpNT/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/postgres -l logfile start
2026-09-28 23:18:39.233 UTC [12154] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 23:18:39.234 UTC [12154] LOG: listening on IPv4 address "127.0.98.20", port 51147
2026-09-28 23:18:39.236 UTC [12154] LOG: listening on Unix socket "/tmp/.s.PGSQL.51147"
2026-09-28 23:18:39.254 UTC [12157] LOG: database system was shut down at 2026-09-28 23:18:38 UTC
2026-09-28 23:18:39.258 UTC [12154] LOG: database system is ready to accept connections
I20260928 23:18:40.127175 392 mini_postgres.cc:96] Postgres bound to 51147
2026-09-28 23:18:40.132 UTC [12165] FATAL: database "slave" does not exist
127.0.98.20:51147 - accepting connections
I20260928 23:18:40.133019 392 mini_ranger.cc:161] Starting Ranger...
I20260928 23:18:40.146399 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 23:18:40.561461 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 23:18:40.561538 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-admin
2026-09-28 23:18:40,832 [I] DB FLAVOR :POSTGRES
2026-09-28 23:18:40,833 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:18:40,833 [I] Checking connection..
2026-09-28 23:18:40,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 23:18:41,057 [I] Checking connection passed.
2026-09-28 23:18:41,057 [I] --------- Verifying version history table ---------
2026-09-28 23:18:41,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:18:41,305 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:18:41,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:18:41,543 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:18:41,543 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 23:18:41,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 23:18:41.786 UTC [12229] WARNING: there is no transaction in progress
2026-09-28 23:18:41,796 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 23:18:41,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:18:42,039 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 23:18:42,039 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:18:42,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 23:18:42,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 23:18:42,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-djkr.c.gcp-upstream.internal','N') ;"
2026-09-28 23:18:42,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:18:43,025 [I] Table x_portal_user does not exist in database ranger
2026-09-28 23:18:43,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:18:43,274 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 23:18:43,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:18:43,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:18:43,737 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 23:18:43,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 23:18:44.604 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.619 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.634 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.648 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.661 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.678 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.726 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.732 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.745 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.754 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.763 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.775 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.783 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.789 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.795 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:44.801 UTC [12337] WARNING: there is no transaction in progress
2026-09-28 23:18:45,335 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 23:18:45,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 23:18:45,573 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 23:18:45,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:18:45,827 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 23:18:45,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:18:46,077 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 23:18:46,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:18:46,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:18:46,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-djkr.c.gcp-upstream.internal';"
2026-09-28 23:18:46,799 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 23:18:46,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 23:18:47,024 [I] DB_PATCHES have already been applied
I20260928 23:18:47.029783 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNOOpNT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 23:18:47.029855 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 23:18:47.032699 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:56023
Sep 28, 2026 11:18:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:18:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:18:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:18:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:18:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:18:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:18:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 23:18:48.020511 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.887s user 0.000s sys 0.008s
Sep 28, 2026 11:19:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:19:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 11:19:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:19:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:19:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 23:19:08.731745 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 23:19:08 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637548, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 23:19:08.803191 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 23:19:08.810905 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 23:19:09.568825 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 23:19:09.568902 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-kms
I20260928 23:19:09.568920 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51147
2026-09-28 23:19:09,714 [I] DB FLAVOR :POSTGRES
2026-09-28 23:19:09,714 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:19:09,714 [I] Checking connection
2026-09-28 23:19:09,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 23:19:09,952 [I] connection success
2026-09-28 23:19:09,952 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 23:19:09,952 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 23:19:09,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 23:19:10,172 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 23:19:10,172 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:19:10,172 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 23:19:10,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51147/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 23:19:10,413 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 23:19:10.822906 392 mini_ranger_kms.cc:310] Created kms service
I20260928 23:19:11.001276 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 23:19:11.040755 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 23:19:11 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637548, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:19:11.399744 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 23:19:11.399844 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNOOpNT/build/release/bin/postgresql.jar:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNOOpNT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 23:19:11.399863 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 23:19:11.403515 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12517
I20260928 23:19:11.403609 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60377
Sep 28, 2026 11:19:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:19:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:19:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:19:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:19:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:19:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 23:19:13.418586 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.615s user 0.003s sys 0.125s
I20260928 23:19:13.418687 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 23:19:13.418730 392 mini_ranger_kms.cc:401] 127.0.98.20:60377/kms/v1/keys
Sep 28 23:19:13 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790637553, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 23:19:13 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637553, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 23:19:14 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637553, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 23:19:16 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:19:16.388454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42297
--ranger_config_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60377/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:16.467312 12568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:16.467527 12568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:16.467571 12568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:16.468755 12568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:19:16.468809 12568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:16.468822 12568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:16.468832 12568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:19:16.468843 12568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:19:16.470454 12568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60377/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42297
--ranger_config_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:16.470669 12568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:16.470887 12568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:16.475700 12574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:16.475646 12576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:16.475638 12573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:16.475714 12568 server_base.cc:1034] running on GCE node
I20260928 23:19:16.476112 12568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:16.476394 12568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:16.477540 12568 hybrid_clock.cc:648] HybridClock initialized: now 1790637556477528 us; error 27 us; skew 500 ppm
Sep 28 23:19:16 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:16.480083 12568 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:19:16.480485 12568 webserver.cc:460] Webserver started at http://127.0.98.62:38559/ using document root <none> and password file <none>
I20260928 23:19:16.480684 12568 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:16.480721 12568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:16.480835 12568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:19:16 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:19:16.532099 12568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "faf8b5bc96984e6d8bbc092920da58a8"
format_stamp: "Formatted at 2026-09-28 23:19:16 on dist-test-slave-djkr"
server_key: "1485352b5f61ee765f9d9b5ea60b4570"
server_key_iv: "bf6615467f7b60e18f35796cd3070c85"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:16.532521 12568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "faf8b5bc96984e6d8bbc092920da58a8"
format_stamp: "Formatted at 2026-09-28 23:19:16 on dist-test-slave-djkr"
server_key: "1485352b5f61ee765f9d9b5ea60b4570"
server_key_iv: "bf6615467f7b60e18f35796cd3070c85"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:16.534257 12568 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 23:19:16.547988 12583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:16.548156 12568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 23:19:16.548238 12568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "faf8b5bc96984e6d8bbc092920da58a8"
format_stamp: "Formatted at 2026-09-28 23:19:16 on dist-test-slave-djkr"
server_key: "1485352b5f61ee765f9d9b5ea60b4570"
server_key_iv: "bf6615467f7b60e18f35796cd3070c85"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:16.548332 12568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:16.570726 12568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:16.572587 12568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:16.572703 12568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:16.576535 12568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42297
I20260928 23:19:16.576582 12635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42297 every 8 connection(s)
I20260928 23:19:16.576906 12568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:19:16.577502 12636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:16.579965 12636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: Bootstrap starting.
I20260928 23:19:16.580542 12636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:16.580780 12636 log.cc:824] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:16.581401 12636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: No bootstrap required, opened a new log
I20260928 23:19:16.583581 12636 raft_consensus.cc:348] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260928 23:19:16.583738 12636 raft_consensus.cc:374] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:16.583760 12636 raft_consensus.cc:729] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf8b5bc96984e6d8bbc092920da58a8, State: Initialized, Role: FOLLOWER
I20260928 23:19:16.583897 12636 consensus_queue.cc:260] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260928 23:19:16.583951 12636 raft_consensus.cc:388] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:19:16.583971 12636 raft_consensus.cc:482] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:19:16.584012 12636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:16.584190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 12568
I20260928 23:19:16.584285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:19:16.584482 392 external_mini_cluster.cc:1428] Setting key 3eaf1f01754bc45c75b7b1748c216f5a
I20260928 23:19:16.584698 12636 raft_consensus.cc:504] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260928 23:19:16.584803 12636 leader_election.cc:302] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faf8b5bc96984e6d8bbc092920da58a8; no voters:
I20260928 23:19:16.584971 12636 leader_election.cc:288] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:19:16.585134 12642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:19:16.585238 12636 sys_catalog.cc:564] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:19:16.585256 12642 raft_consensus.cc:686] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 1 LEADER]: Becoming Leader. State: Replica: faf8b5bc96984e6d8bbc092920da58a8, State: Running, Role: LEADER
I20260928 23:19:16.585428 12642 consensus_queue.cc:237] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260928 23:19:16.586052 12639 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faf8b5bc96984e6d8bbc092920da58a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } } }
I20260928 23:19:16.586061 12643 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faf8b5bc96984e6d8bbc092920da58a8. Latest consensus state: current_term: 1 leader_uuid: "faf8b5bc96984e6d8bbc092920da58a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } } }
I20260928 23:19:16.586278 12643 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:16.586218 12639 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:16.586477 12653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:19:16.586721 12636 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:19:16.586922 12653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:19:16.591471 12653 catalog_manager.cc:1347] Generated new cluster ID: 5abd29cc0afc4e8d960a1565070804d5
I20260928 23:19:16.591538 12653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:19:16 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:16.599320 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:16.584941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.0.98.62:42297)
0928 23:19:16.585551 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:16.585561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:16.585589 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:19:16.585707 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:16.585715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:16.585778 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:16.585921 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:16.587044 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:16.590612 (+ 3568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:16.591742 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:16.592365 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:16.592473 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:16.595271 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:16.595276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:16.595276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:16.595288 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:16.597295 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:16.597684 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:16.597688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:16.597688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:16.597782 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:16.597999 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:16.598000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:16.598000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:16.598304 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:16.598460 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:16.598974 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 23:19:16.599099 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":62,"threads_started":1}
I20260928 23:19:16.604067 12653 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:19:16.604797 12653 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:19:16.608830 12653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: Generated new TSK 0
I20260928 23:19:16.608999 12653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:19:17.288066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:17.370872 12681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:17.371075 12681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:17.371106 12681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:17.372285 12681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:17.372325 12681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:17.372359 12681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:19:17.373778 12681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:17.374006 12681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:17.374218 12681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:17.378804 12687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:17.378804 12689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:17.379189 12681 server_base.cc:1034] running on GCE node
W20260928 23:19:17.379125 12686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:17.379460 12681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:17.379693 12681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:17.380827 12681 hybrid_clock.cc:648] HybridClock initialized: now 1790637557380811 us; error 27 us; skew 500 ppm
Sep 28 23:19:17 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:17.389035 12681 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:19:17.389489 12681 webserver.cc:460] Webserver started at http://127.0.98.1:44833/ using document root <none> and password file <none>
I20260928 23:19:17.389716 12681 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:17.389763 12681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:17.389891 12681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:19:17 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:19:17.404285 12681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "8fb6096151204d5f892f9b42b27fb1c6"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "c71fe425b1cfec173cc859573137e99e"
server_key_iv: "bcfb04c61f4bd3d96f4b43381f894d4a"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:17.404716 12681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8fb6096151204d5f892f9b42b27fb1c6"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "c71fe425b1cfec173cc859573137e99e"
server_key_iv: "bcfb04c61f4bd3d96f4b43381f894d4a"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:17.406461 12681 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:19:17.420423 12696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:17.420641 12681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:19:17.420745 12681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "8fb6096151204d5f892f9b42b27fb1c6"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "c71fe425b1cfec173cc859573137e99e"
server_key_iv: "bcfb04c61f4bd3d96f4b43381f894d4a"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:17.420840 12681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:17.435050 12681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:17.437314 12681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:17.437444 12681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:17.437680 12681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:17.438060 12681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:17.438133 12681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:17.438186 12681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:17.438226 12681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:17.444868 12681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37727
I20260928 23:19:17.445132 12809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37727 every 8 connection(s)
I20260928 23:19:17.445331 12681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:19:17.446448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 12681
I20260928 23:19:17.446547 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:19:17.446724 392 external_mini_cluster.cc:1428] Setting key ed35ce0f9be5c63d16e2737d1b1dc3b4
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 28 23:19:17 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:17.463178 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:17.446060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42297 (local address 127.0.98.1:53389)
0928 23:19:17.447095 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:17.447142 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:17.449283 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:17.454895 (+ 5612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:17.454900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:17.455132 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:17.455333 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:17.455338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.456062 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.456063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:17.456907 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:17.456911 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.457419 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.457421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:17.457573 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:17.458146 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:17.458156 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:17.459714 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:17.461681 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:17.461687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:17.461689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:17.461980 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:17.462194 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:17.462195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:17.462196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:17.462329 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:17.462627 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:17.462634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:17.462812 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 23:19:17.462892 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:19:17.463011 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":69,"threads_started":1}
I20260928 23:19:17.463662 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:17.446909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53389 (local address 127.0.98.62:42297)
0928 23:19:17.454625 (+ 7716us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:17.454627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:17.454638 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:19:17.454665 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:17.454666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:17.454680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:17.454788 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:17.455439 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.455980 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.457005 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.457303 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.457805 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:17.459820 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:17.459822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:17.459822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:17.459831 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:17.461584 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:17.462064 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:17.462065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:17.462066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:17.462127 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:17.462410 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:17.462411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:17.462411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:17.462550 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:17.463435 (+ 885us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:17.463476 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 23:19:17.463573 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7682,"thread_start_us":76,"threads_started":1}
I20260928 23:19:17.464217 12810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42297
I20260928 23:19:17.464308 12810 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:17.464483 12810 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260928 23:19:17.465065 12600 ts_manager.cc:194] Registered new tserver with Master: 8fb6096151204d5f892f9b42b27fb1c6 (127.0.98.1:37727)
I20260928 23:19:17.465900 12600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:19:17.476035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:17.591744 12818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:17.591948 12818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:17.591979 12818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:17.593255 12818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:17.593297 12818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:17.593333 12818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:19:17.594831 12818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:17.595067 12818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:17.595273 12818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:17.599974 12825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:17.600102 12826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:17.600003 12828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:17.600488 12818 server_base.cc:1034] running on GCE node
I20260928 23:19:17.600689 12818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:17.600907 12818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:17.604139 12818 hybrid_clock.cc:648] HybridClock initialized: now 1790637557604121 us; error 32 us; skew 500 ppm
Sep 28 23:19:17 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:17.606725 12818 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:19:17.607185 12818 webserver.cc:460] Webserver started at http://127.0.98.2:44643/ using document root <none> and password file <none>
I20260928 23:19:17.607402 12818 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:17.607450 12818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:17.607587 12818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:19:17 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:19:17.618556 12818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "8f487700fd534a1aa288ba5d8db77c16"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "8594fc90fc266e96ae10822796eef495"
server_key_iv: "d5b62342f6fdc34884727bd2430b71be"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:17.619093 12818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8f487700fd534a1aa288ba5d8db77c16"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "8594fc90fc266e96ae10822796eef495"
server_key_iv: "d5b62342f6fdc34884727bd2430b71be"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:17.629137 12818 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.000s
I20260928 23:19:17.644407 12835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:17.644639 12818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:17.644726 12818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "8f487700fd534a1aa288ba5d8db77c16"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "8594fc90fc266e96ae10822796eef495"
server_key_iv: "d5b62342f6fdc34884727bd2430b71be"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:17.644840 12818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 23:19:17 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790637557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:17.657833 12818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:17.660358 12818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:17.660561 12818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:17.660805 12818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:17.661230 12818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:17.661326 12818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:17.661401 12818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:17.661459 12818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:17.669895 12818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44555
I20260928 23:19:17.670392 12818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:19:17.670933 12948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44555 every 8 connection(s)
I20260928 23:19:17.675462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 12818
I20260928 23:19:17.675583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:19:17.675757 392 external_mini_cluster.cc:1428] Setting key afbed6bad60c44bc843aa80dbcc4debf
Sep 28 23:19:17 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:17.682296 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:17.671651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55543 (local address 127.0.98.62:42297)
0928 23:19:17.671701 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:17.671703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:17.672313 (+ 610us) server_negotiation.cc:407] Connection header received
0928 23:19:17.673026 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:17.673027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:17.673040 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:17.673070 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:17.673914 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.674652 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.676279 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.676592 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.676637 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:17.679063 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:17.679065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:17.679065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:17.679073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:17.680668 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:17.681150 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:17.681151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:17.681152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:17.681209 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:17.681484 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:17.681485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:17.681486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:17.681620 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:17.681663 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:17.682108 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 23:19:17.682199 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:19:17.683032 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:17.671765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42297 (local address 127.0.98.2:55543)
0928 23:19:17.672183 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:17.672226 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:17.672896 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:17.673259 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:17.673263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:17.673502 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:17.673782 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:17.673789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.674764 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.674765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:17.676178 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:17.676180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.676740 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.676741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:17.676887 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:17.677426 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:17.677436 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:17.678959 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:17.680761 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:17.680767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:17.680770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:17.681062 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:17.681275 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:17.681276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:17.681276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:17.681406 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:17.681709 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:17.681716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:17.681908 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 23:19:17.682623 (+ 715us) client_negotiation.cc:240] Negotiation successful
0928 23:19:17.682789 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":84,"threads_started":1}
I20260928 23:19:17.683686 12949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42297
I20260928 23:19:17.683777 12949 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:17.683969 12949 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260928 23:19:17.684371 12600 ts_manager.cc:194] Registered new tserver with Master: 8f487700fd534a1aa288ba5d8db77c16 (127.0.98.2:44555)
I20260928 23:19:17.684883 12600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55543
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:19:17.728289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:17.847767 12954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:17.848095 12954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:17.848173 12954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:17.849854 12954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:17.849960 12954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:17.850042 12954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:19:17.852367 12954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:17.852803 12954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:17.853111 12954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:17.860364 12960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:17.860364 12961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:17.860723 12963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:17.861557 12954 server_base.cc:1034] running on GCE node
I20260928 23:19:17.861759 12954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:17.862013 12954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:17.865293 12954 hybrid_clock.cc:648] HybridClock initialized: now 1790637557865268 us; error 37 us; skew 500 ppm
Sep 28 23:19:17 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:17.868366 12954 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:19:17.868861 12954 webserver.cc:460] Webserver started at http://127.0.98.3:35651/ using document root <none> and password file <none>
I20260928 23:19:17.869097 12954 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:17.869151 12954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:17.869273 12954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:19:17 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:19:17.888758 12954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "47e480289bc2458ca164ac2153e692f4"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "a913cd0a2bed13ff820d682d36583d53"
server_key_iv: "6cb24503851e3e7531f537f3ed28dcd0"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:17.889305 12954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47e480289bc2458ca164ac2153e692f4"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "a913cd0a2bed13ff820d682d36583d53"
server_key_iv: "6cb24503851e3e7531f537f3ed28dcd0"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:17.891925 12954 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:19:17.919540 12970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:17.919821 12954 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 23:19:17.919924 12954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "47e480289bc2458ca164ac2153e692f4"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "a913cd0a2bed13ff820d682d36583d53"
server_key_iv: "6cb24503851e3e7531f537f3ed28dcd0"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:17.920024 12954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:17.938346 12954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:17.940078 12954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:17.940204 12954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:17.940444 12954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:17.940779 12954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:17.940816 12954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:17.940846 12954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:17.940891 12954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:17.947294 12954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37587
I20260928 23:19:17.947715 12954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:19:17.952539 13083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37587 every 8 connection(s)
Sep 28 23:19:17 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:17.956992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 12954
I20260928 23:19:17.957085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:19:17.957278 392 external_mini_cluster.cc:1428] Setting key 8339e72001c739d5a82742071c721779
I20260928 23:19:17.959417 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:17.948426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45965 (local address 127.0.98.62:42297)
0928 23:19:17.948482 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:17.948484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:17.951797 (+ 3313us) server_negotiation.cc:407] Connection header received
0928 23:19:17.952342 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:17.952343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:17.952355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:17.952414 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:17.953098 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.953589 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.954547 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.954736 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.954790 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:17.956723 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:17.956726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:17.956726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:17.956734 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:17.958240 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:17.958579 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:17.958580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:17.958580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:17.958636 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:17.958836 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:17.958836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:17.958837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:17.958948 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:17.959000 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:17.959257 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 23:19:17.959331 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:19:17.959614 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:17.948418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42297 (local address 127.0.98.3:45965)
0928 23:19:17.951663 (+ 3245us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:17.951701 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:17.952240 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:17.952564 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:17.952569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:17.952795 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:17.953001 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:17.953006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.953672 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.953673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:17.954464 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:17.954466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:17.954814 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:17.954816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:17.954925 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:17.955422 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:17.955438 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:17.956636 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:17.958300 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:17.958308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:17.958310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:17.958514 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:17.958686 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:17.958687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:17.958687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:17.958782 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:17.959000 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:17.959006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:17.959141 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 23:19:17.959228 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:19:17.959364 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3207,"thread_start_us":77,"threads_started":1}
I20260928 23:19:17.960103 13084 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42297
I20260928 23:19:17.960193 13084 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:17.960357 13084 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260928 23:19:17.960687 12600 ts_manager.cc:194] Registered new tserver with Master: 47e480289bc2458ca164ac2153e692f4 (127.0.98.3:37587)
I20260928 23:19:17.961042 12600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45965
I20260928 23:19:17.975345 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:19:17.976213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 12681
I20260928 23:19:17.981475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 12818
I20260928 23:19:17.987623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 12954
I20260928 23:19:17.992327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 12568
I20260928 23:19:17.998099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42297
--webserver_interface=127.0.98.62
--webserver_port=38559
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42297
--ranger_config_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60377/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60377/kms with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:18.080539 13087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:18.080724 13087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:18.080744 13087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:18.081823 13087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:19:18.081862 13087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:18.081874 13087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:18.081897 13087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:19:18.081919 13087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:19:18.083439 13087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60377/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42297
--ranger_config_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:18.083662 13087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:18.083853 13087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:18.088233 13092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:18.088240 13093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:18.088291 13087 server_base.cc:1034] running on GCE node
W20260928 23:19:18.088207 13095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:18.088595 13087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:18.088894 13087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:18.090030 13087 hybrid_clock.cc:648] HybridClock initialized: now 1790637558090015 us; error 30 us; skew 500 ppm
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:18.092280 13087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:19:18.092692 13087 webserver.cc:460] Webserver started at http://127.0.98.62:38559/ using document root <none> and password file <none>
I20260928 23:19:18.092885 13087 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:18.092939 13087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:19:18.138819 13087 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:18.139614 13103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:18.139779 13087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:18.139881 13087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "faf8b5bc96984e6d8bbc092920da58a8"
format_stamp: "Formatted at 2026-09-28 23:19:16 on dist-test-slave-djkr"
server_key: "1485352b5f61ee765f9d9b5ea60b4570"
server_key_iv: "bf6615467f7b60e18f35796cd3070c85"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:18.140318 13087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:18.155383 13087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:18.157058 13087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:18.157161 13087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:18.161082 13087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42297
I20260928 23:19:18.161116 13155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42297 every 8 connection(s)
I20260928 23:19:18.161437 13087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:19:18.162576 13156 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:19:18.163170 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: Bootstrap starting.
I20260928 23:19:18.163815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 13087
I20260928 23:19:18.164084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37727
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42297
--webserver_port=44833
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60377/kms with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:19:18.171072 13156 log.cc:824] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:18.172704 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:19:18.172935 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: Bootstrap complete.
I20260928 23:19:18.175012 13156 raft_consensus.cc:348] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260928 23:19:18.175230 13156 raft_consensus.cc:729] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf8b5bc96984e6d8bbc092920da58a8, State: Initialized, Role: FOLLOWER
I20260928 23:19:18.175374 13156 consensus_queue.cc:260] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260928 23:19:18.175431 13156 raft_consensus.cc:388] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:19:18.175479 13156 raft_consensus.cc:482] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:19:18.175522 13156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:19:18.176499 13156 raft_consensus.cc:504] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260928 23:19:18.176584 13156 leader_election.cc:302] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faf8b5bc96984e6d8bbc092920da58a8; no voters:
I20260928 23:19:18.176719 13156 leader_election.cc:288] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:19:18.176774 13160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:19:18.176952 13156 sys_catalog.cc:564] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:19:18.176966 13160 raft_consensus.cc:686] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [term 2 LEADER]: Becoming Leader. State: Replica: faf8b5bc96984e6d8bbc092920da58a8, State: Running, Role: LEADER
I20260928 23:19:18.177191 13160 consensus_queue.cc:237] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260928 23:19:18.177484 13161 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "faf8b5bc96984e6d8bbc092920da58a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } } }
I20260928 23:19:18.177549 13161 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:18.177904 13168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:19:18.178241 13162 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faf8b5bc96984e6d8bbc092920da58a8. Latest consensus state: current_term: 2 leader_uuid: "faf8b5bc96984e6d8bbc092920da58a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8b5bc96984e6d8bbc092920da58a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } } }
I20260928 23:19:18.178354 13162 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:18.178501 13168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:19:18.178645 13168 catalog_manager.cc:1259] Loaded cluster ID: 5abd29cc0afc4e8d960a1565070804d5
I20260928 23:19:18.178704 13168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:19:18.178805 13156 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:19:18.182474 13168 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:19:18.183015 13168 catalog_manager.cc:5879] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: Loaded TSK: 0
I20260928 23:19:18.183292 13168 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:19:18.271471 13158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:18.271759 13158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:18.271788 13158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:18.273389 13158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:18.273476 13158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:18.273543 13158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:19:18.275629 13158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44833
--tserver_master_addrs=127.0.98.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:18.275961 13158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:18.276198 13158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:18.281994 13193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:18.282245 13194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:18.282560 13158 server_base.cc:1034] running on GCE node
W20260928 23:19:18.282617 13196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:18.282867 13158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:18.283128 13158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:18.284297 13158 hybrid_clock.cc:648] HybridClock initialized: now 1790637558284281 us; error 28 us; skew 500 ppm
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:18.287292 13158 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:19:18.287819 13158 webserver.cc:460] Webserver started at http://127.0.98.1:44833/ using document root <none> and password file <none>
I20260928 23:19:18.288066 13158 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:18.288146 13158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:19:18.309152 13158 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 23:19:18.313469 13204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:18.313684 13158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260928 23:19:18.313781 13158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "8fb6096151204d5f892f9b42b27fb1c6"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "c71fe425b1cfec173cc859573137e99e"
server_key_iv: "bcfb04c61f4bd3d96f4b43381f894d4a"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:18.314208 13158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:18.329007 13158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:18.331066 13158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:18.331199 13158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:18.331406 13158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:18.331751 13158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:18.331786 13158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:18.331815 13158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:18.331861 13158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:18.337405 13158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37727
I20260928 23:19:18.337814 13158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:19:18.339762 13317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37727 every 8 connection(s)
I20260928 23:19:18.340065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 13158
I20260928 23:19:18.340318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44555
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42297
--webserver_port=44643
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60377/kms with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:18.352306 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:18.340364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42297 (local address 127.0.98.1:53135)
0928 23:19:18.340608 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:18.340648 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:18.341196 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:18.341849 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:18.341853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:18.342085 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:18.342357 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:18.342365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.343528 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.343529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:18.344743 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:18.344746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.345604 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.345606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:18.345752 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:18.346219 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:18.346228 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:18.347811 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:18.349929 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:18.349935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:18.349937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:18.350212 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:18.350493 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:18.350494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:18.350494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:18.350630 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:18.351659 (+ 1029us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:18.351665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:18.351824 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 23:19:18.351938 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:19:18.352087 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
I20260928 23:19:18.352316 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:18.340465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53135 (local address 127.0.98.62:42297)
0928 23:19:18.340642 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:18.340652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:18.340932 (+ 280us) server_negotiation.cc:407] Connection header received
0928 23:19:18.341505 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:18.341514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:18.341578 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:18.341706 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:18.342486 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.343425 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.344862 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.345502 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.345622 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:18.347926 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:18.347930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:18.347931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:18.347944 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:18.349839 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:18.350302 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:18.350305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:18.350305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:18.350417 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:18.350700 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:18.350701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:18.350701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:18.351541 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:18.351661 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:18.351994 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 23:19:18.352120 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260928 23:19:18.353474 13318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42297
I20260928 23:19:18.353570 13318 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:18.353744 13318 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
W20260928 23:19:18.359692 13318 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42297 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:19:18.359767 13318 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:18.359829 13318 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260928 23:19:18.360240 13318 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:18.360311 13318 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
W20260928 23:19:18.360618 13318 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:19:18.440014 13322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:18.440271 13322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:18.440310 13322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:18.441979 13322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:18.442061 13322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:18.442121 13322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:19:18.444188 13322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44643
--tserver_master_addrs=127.0.98.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:18.444551 13322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:18.444854 13322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:18.450011 13327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:18.450026 13328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:18.450230 13330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:18.450675 13322 server_base.cc:1034] running on GCE node
I20260928 23:19:18.450877 13322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:18.451124 13322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:18.452280 13322 hybrid_clock.cc:648] HybridClock initialized: now 1790637558452261 us; error 35 us; skew 500 ppm
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:18.455272 13322 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:19:18.455763 13322 webserver.cc:460] Webserver started at http://127.0.98.2:44643/ using document root <none> and password file <none>
I20260928 23:19:18.455984 13322 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:18.456048 13322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:19:18.485001 13322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:18.485757 13338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:18.485935 13322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:18.486033 13322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "8f487700fd534a1aa288ba5d8db77c16"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "8594fc90fc266e96ae10822796eef495"
server_key_iv: "d5b62342f6fdc34884727bd2430b71be"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:18.486466 13322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:18.507752 13322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:18.509925 13322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:18.510064 13322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:18.510321 13322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:18.510677 13322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:18.510723 13322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:18.510753 13322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:18.510818 13322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:18.517800 13322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44555
I20260928 23:19:18.517840 13451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44555 every 8 connection(s)
I20260928 23:19:18.518266 13322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:19:18.519116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 13322
I20260928 23:19:18.519384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60377/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37587
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42297
--webserver_port=35651
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60377/kms with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:18.528555 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:18.519618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53121 (local address 127.0.98.62:42297)
0928 23:19:18.519665 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:18.519667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:18.519679 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:19:18.519892 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:18.519894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:18.519909 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:18.519956 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:18.520480 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.521128 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.522275 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.522582 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.522624 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:18.525230 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:18.525231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:18.525232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:18.525240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:18.526892 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:18.527425 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:18.527427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:18.527427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:18.527494 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:18.527809 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:18.527810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:18.527810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:18.527943 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:18.527984 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:18.528367 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 23:19:18.528447 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:19:18.529263 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:18.519000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42297 (local address 127.0.98.2:53121)
0928 23:19:18.519220 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:18.519260 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:18.519794 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:18.520040 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:18.520044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:18.520214 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:18.520397 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:18.520401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.521214 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.521215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:18.522182 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:18.522185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.522719 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.522720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:18.522847 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:18.523361 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:18.523372 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:18.525097 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:18.527013 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:18.527019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:18.527021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:18.527321 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:18.527600 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:18.527601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:18.527601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:18.527732 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:18.528031 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:18.528037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:18.528207 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 23:19:18.528871 (+ 664us) client_negotiation.cc:240] Negotiation successful
0928 23:19:18.529032 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
I20260928 23:19:18.529886 13452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42297
I20260928 23:19:18.529975 13452 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:18.530151 13452 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
W20260928 23:19:18.530794 13452 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42297 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:19:18.530853 13452 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:18.530905 13452 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260928 23:19:18.531296 13452 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:18.531359 13452 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
W20260928 23:19:18.531652 13452 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:19:18.617743 13455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:18.617924 13455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:18.617952 13455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:18.619307 13455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:18.619346 13455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:18.619380 13455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:19:18.621248 13455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60377/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35651
--tserver_master_addrs=127.0.98.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:18.621467 13455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:18.621673 13455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:18.626257 13463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:18.626256 13461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:18.626678 13455 server_base.cc:1034] running on GCE node
W20260928 23:19:18.627692 13460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:18.627952 13455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:18.628201 13455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:18.629349 13455 hybrid_clock.cc:648] HybridClock initialized: now 1790637558629327 us; error 36 us; skew 500 ppm
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:18.631811 13455 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:19:18.632230 13455 webserver.cc:460] Webserver started at http://127.0.98.3:35651/ using document root <none> and password file <none>
I20260928 23:19:18.632428 13455 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:18.632488 13455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:19:18.658674 13455 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260928 23:19:18.660369 13472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:18.660640 13455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:18.660732 13455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "47e480289bc2458ca164ac2153e692f4"
format_stamp: "Formatted at 2026-09-28 23:19:17 on dist-test-slave-djkr"
server_key: "a913cd0a2bed13ff820d682d36583d53"
server_key_iv: "6cb24503851e3e7531f537f3ed28dcd0"
server_key_version: "kuduclusterkey@0"
I20260928 23:19:18.661202 13455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:18.673372 13455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:18.676548 13455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:18.676749 13455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:18.677009 13455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:18.677430 13455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:18.677520 13455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:18.677591 13455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:18.677644 13455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:18.683921 13455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37587
I20260928 23:19:18.684415 13455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:19:18.686275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 13455
I20260928 23:19:18.695947 13585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37587 every 8 connection(s)
Sep 28 23:19:18 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:18.705938 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:18.686676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.0.98.62:42297)
0928 23:19:18.686722 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:18.686724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:18.696843 (+ 10119us) server_negotiation.cc:407] Connection header received
0928 23:19:18.696875 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:18.696876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:18.696889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:18.696932 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:18.697788 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.698570 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.699735 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.700025 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.700077 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:18.701522 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:18.701524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:18.701524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:18.701531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:18.703283 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:18.704513 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:18.704514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:18.704515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:18.704574 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:18.705021 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:18.705022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:18.705023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:18.705387 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:18.705547 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:18.705749 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 23:19:18.705842 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:19:18.707629 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:18.697035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46513 (local address 127.0.98.62:42297)
0928 23:19:18.697935 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:18.697936 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:18.697945 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:19:18.697965 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:18.697966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:18.697976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:18.699065 (+ 1089us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:18.699102 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.699884 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.701293 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.701602 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.702016 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:18.704459 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:18.704460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:18.704460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:18.704466 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:18.705942 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:18.706407 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:18.706407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:18.706408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:18.706506 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:18.706888 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:18.706889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:18.706889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:18.707035 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:18.707100 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:18.707437 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 23:19:18.707518 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":51,"threads_started":1}
I20260928 23:19:18.707926 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:18.696702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42297 (local address 127.0.98.3:46513)
0928 23:19:18.696978 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:18.697138 (+ 160us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:18.697761 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:18.698153 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:18.698157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:18.698669 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:18.698972 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:18.698979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.699983 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.699984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:18.701204 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:18.701206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:18.701691 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:18.701692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:18.701820 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:18.702240 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:18.702250 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:18.704366 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:18.706032 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:18.706039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:18.706041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:18.706324 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:18.706576 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:18.706577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:18.706577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:18.706808 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:18.707106 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:18.707112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:18.707285 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 23:19:18.707407 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 23:19:18.707580 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":84,"threads_started":1}
I20260928 23:19:18.708305 13586 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42297
I20260928 23:19:18.708451 13586 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:18.708626 13586 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
W20260928 23:19:18.709321 13586 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42297 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:19:18.709400 13586 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:18.709470 13586 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260928 23:19:18.709805 13586 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:18.709877 13586 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
W20260928 23:19:18.710193 13586 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 23:19:19 dist-test-slave-djkr krb5kdc[12124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790637559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:19.360807 13318 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:19.360987 13318 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260928 23:19:19.361357 13119 ts_manager.cc:194] Registered new tserver with Master: 8fb6096151204d5f892f9b42b27fb1c6 (127.0.98.1:37727)
I20260928 23:19:19.362452 13119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53135
I20260928 23:19:19.531810 13452 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:19.531970 13452 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260928 23:19:19.532275 13119 ts_manager.cc:194] Registered new tserver with Master: 8f487700fd534a1aa288ba5d8db77c16 (127.0.98.2:44555)
I20260928 23:19:19.532799 13119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53121
Sep 28 23:19:19 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:19:19.710364 13586 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:19.710516 13586 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260928 23:19:19.710829 13120 ts_manager.cc:194] Registered new tserver with Master: 47e480289bc2458ca164ac2153e692f4 (127.0.98.3:37587)
I20260928 23:19:19.711339 13120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46513
I20260928 23:19:19.717389 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:19:19.730738 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.723861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.0.98.62:42297)
0928 23:19:19.724158 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:19.724160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:19.724166 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:19:19.724188 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:19.724189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:19.724200 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:19.724264 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:19.724972 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.725775 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.726797 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.727103 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.727167 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:19.728420 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:19.728421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:19.728422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:19.728429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:19.729731 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:19.730052 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:19.730053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:19.730054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:19.730107 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:19.730265 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:19.730266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:19.730266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:19.730369 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:19.730452 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:19.730556 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 23:19:19.730632 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":60,"threads_started":1}
I20260928 23:19:19.732496 13120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:19:19.734120 13120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:19:19.741134 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.736563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37587 (local address 127.0.0.1:40066)
0928 23:19:19.736912 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:19.736953 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:19.737030 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:19.737318 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:19.737322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:19.737324 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:19.737389 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:19.737390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.738260 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.738261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:19.739856 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:19.739858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.740851 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.740851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:19.740926 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:19.740941 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 23:19:19.741005 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:19:19.741056 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":8,"thread_start_us":77,"threads_started":1}
I20260928 23:19:19.741617 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.736808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.0.98.3:37587)
0928 23:19:19.736970 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:19.736976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:19.737075 (+ 99us) server_negotiation.cc:407] Connection header received
0928 23:19:19.737112 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:19.737114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:19.737131 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:19.737202 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:19.737465 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.738190 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.739936 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.740669 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.740739 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:19.740821 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:19.741244 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 23:19:19.741290 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":47,"threads_started":1}
I20260928 23:19:19.742295 13504 tablet_service.cc:1467] Processing CreateTablet for tablet d1af877d8b70480a8bf7f4c130fb27fc (DEFAULT_TABLE table=test-table [id=6730d83243ed4115a1df37c50b8ce403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:19.742533 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.736565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44555 (local address 127.0.0.1:36680)
0928 23:19:19.737123 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:19.737131 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:19.737169 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:19.738450 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:19.738450 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:19.738452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:19.738484 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:19.738485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.739644 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.739644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:19.740685 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:19.740688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.742266 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.742267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:19.742324 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:19.742328 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:19:19.742372 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:19:19.742408 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":80,"threads_started":1}
I20260928 23:19:19.742668 13504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1af877d8b70480a8bf7f4c130fb27fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:19.743707 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.736668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.0.98.2:44555)
0928 23:19:19.737052 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:19.737056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:19.737274 (+ 218us) server_negotiation.cc:407] Connection header received
0928 23:19:19.738288 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:19.738291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:19.738309 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:19.738376 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:19.738550 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.739570 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.740765 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.742100 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.742164 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:19.742233 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:19.743595 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0928 23:19:19.743635 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":57,"threads_started":1}
I20260928 23:19:19.745833 13386 tablet_service.cc:1467] Processing CreateTablet for tablet d1af877d8b70480a8bf7f4c130fb27fc (DEFAULT_TABLE table=test-table [id=6730d83243ed4115a1df37c50b8ce403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:19.746160 13386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1af877d8b70480a8bf7f4c130fb27fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:19.748210 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.736786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37727 (local address 127.0.0.1:53658)
0928 23:19:19.737530 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:19.737538 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:19.737572 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:19.744702 (+ 7130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:19.744703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:19.744705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:19.744742 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:19.744743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.745940 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.745941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:19.746968 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:19.746971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.747983 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.747984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:19.748053 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:19.748056 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:19:19.748103 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:19:19.748139 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":64,"threads_started":1}
I20260928 23:19:19.748494 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.736877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.0.98.1:37727)
0928 23:19:19.744471 (+ 7594us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:19.744476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:19.744488 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:19:19.744521 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:19.744523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:19.744539 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:19.744621 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:19.744829 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.745847 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.747072 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:19.747818 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:19.747887 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:19.747950 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:19.748369 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 23:19:19.748412 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7556,"thread_start_us":40,"threads_started":1}
I20260928 23:19:19.749066 13252 tablet_service.cc:1467] Processing CreateTablet for tablet d1af877d8b70480a8bf7f4c130fb27fc (DEFAULT_TABLE table=test-table [id=6730d83243ed4115a1df37c50b8ce403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:19.749344 13252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1af877d8b70480a8bf7f4c130fb27fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:19.753291 13621 tablet_bootstrap.cc:492] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: Bootstrap starting.
I20260928 23:19:19.753904 13621 tablet_bootstrap.cc:654] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:19.754175 13621 log.cc:824] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:19.754452 13620 tablet_bootstrap.cc:492] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6: Bootstrap starting.
I20260928 23:19:19.755004 13620 tablet_bootstrap.cc:654] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:19.755264 13620 log.cc:824] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:19.755672 13619 tablet_bootstrap.cc:492] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: Bootstrap starting.
I20260928 23:19:19.755988 13621 tablet_bootstrap.cc:492] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: No bootstrap required, opened a new log
I20260928 23:19:19.756057 13621 ts_tablet_manager.cc:1397] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:19:19.756260 13619 tablet_bootstrap.cc:654] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:19.756541 13619 log.cc:824] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:19.758571 13621 raft_consensus.cc:348] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 } } peers { permanent_uuid: "47e480289bc2458ca164ac2153e692f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } } peers { permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 } }
I20260928 23:19:19.758724 13621 raft_consensus.cc:374] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:19.758751 13621 raft_consensus.cc:729] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f487700fd534a1aa288ba5d8db77c16, State: Initialized, Role: FOLLOWER
I20260928 23:19:19.758867 13621 consensus_queue.cc:260] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 } } peers { permanent_uuid: "47e480289bc2458ca164ac2153e692f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } } peers { permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 } }
I20260928 23:19:19.759085 13621 ts_tablet_manager.cc:1428] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:19:19.759182 13452 heartbeater.cc:503] Master 127.0.98.62:42297 was elected leader, sending a full tablet report...
I20260928 23:19:19.760416 13620 tablet_bootstrap.cc:492] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6: No bootstrap required, opened a new log
I20260928 23:19:19.760490 13620 ts_tablet_manager.cc:1397] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 23:19:19.762954 13620 raft_consensus.cc:348] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 } } peers { permanent_uuid: "47e480289bc2458ca164ac2153e692f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } } peers { permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 } }
I20260928 23:19:19.763106 13620 raft_consensus.cc:374] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:19.763141 13620 raft_consensus.cc:729] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb6096151204d5f892f9b42b27fb1c6, State: Initialized, Role: FOLLOWER
I20260928 23:19:19.763250 13620 consensus_queue.cc:260] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 } } peers { permanent_uuid: "47e480289bc2458ca164ac2153e692f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } } peers { permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 } }
I20260928 23:19:19.763464 13620 ts_tablet_manager.cc:1428] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:19:19.763592 13318 heartbeater.cc:503] Master 127.0.98.62:42297 was elected leader, sending a full tablet report...
I20260928 23:19:19.765388 13619 tablet_bootstrap.cc:492] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: No bootstrap required, opened a new log
I20260928 23:19:19.765451 13619 ts_tablet_manager.cc:1397] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260928 23:19:19.767890 13619 raft_consensus.cc:348] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 } } peers { permanent_uuid: "47e480289bc2458ca164ac2153e692f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } } peers { permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 } }
I20260928 23:19:19.768054 13619 raft_consensus.cc:374] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:19.768090 13619 raft_consensus.cc:729] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e480289bc2458ca164ac2153e692f4, State: Initialized, Role: FOLLOWER
I20260928 23:19:19.768187 13619 consensus_queue.cc:260] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 } } peers { permanent_uuid: "47e480289bc2458ca164ac2153e692f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } } peers { permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 } }
I20260928 23:19:19.768421 13619 ts_tablet_manager.cc:1428] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:19:19.768543 13586 heartbeater.cc:503] Master 127.0.98.62:42297 was elected leader, sending a full tablet report...
W20260928 23:19:19.769580 13453 tablet.cc:2369] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 23:19:19 dist-test-slave-djkr krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260928 23:19:19.841236 13319 tablet.cc:2369] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:19:19.947338 13587 tablet.cc:2369] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:19:19.998148 13627 raft_consensus.cc:482] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:19:19.998309 13627 raft_consensus.cc:504] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 } } peers { permanent_uuid: "47e480289bc2458ca164ac2153e692f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } } peers { permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 } }
I20260928 23:19:19.998765 13627 leader_election.cc:288] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f487700fd534a1aa288ba5d8db77c16 (127.0.98.2:44555), 8fb6096151204d5f892f9b42b27fb1c6 (127.0.98.1:37727)
I20260928 23:19:20.009042 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.999189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34813 (local address 127.0.98.1:37727)
0928 23:19:19.999229 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:19.999231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:19.999408 (+ 177us) server_negotiation.cc:407] Connection header received
0928 23:19:19.999505 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:19.999506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:19.999518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:19.999565 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:19.999784 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:20.000519 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:20.003374 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:20.004003 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:20.004051 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:20.004064 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:20.008904 (+ 4840us) server_negotiation.cc:299] Negotiation successful
0928 23:19:20.008962 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:19:20.009521 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.998968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37727 (local address 127.0.98.3:34813)
0928 23:19:19.999344 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:19.999355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:19.999435 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:19.999630 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:19.999631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:19.999633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:19.999721 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:19.999723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:20.000619 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:20.000620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:20.003264 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:20.003268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:20.008709 (+ 5441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:20.008710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:20.008797 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:20.008802 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:19:20.009366 (+ 564us) client_negotiation.cc:240] Negotiation successful
0928 23:19:20.009421 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":99,"threads_started":1}
I20260928 23:19:20.010068 13272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1af877d8b70480a8bf7f4c130fb27fc" candidate_uuid: "47e480289bc2458ca164ac2153e692f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb6096151204d5f892f9b42b27fb1c6" is_pre_election: true
I20260928 23:19:20.010247 13272 raft_consensus.cc:2454] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e480289bc2458ca164ac2153e692f4 in term 0.
I20260928 23:19:20.010519 13474 leader_election.cc:302] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e480289bc2458ca164ac2153e692f4, 8fb6096151204d5f892f9b42b27fb1c6; no voters:
I20260928 23:19:20.010669 13627 raft_consensus.cc:2781] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:19:20.010706 13627 raft_consensus.cc:482] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:19:20.010726 13627 raft_consensus.cc:3037] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:20.011631 13627 raft_consensus.cc:504] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 } } peers { permanent_uuid: "47e480289bc2458ca164ac2153e692f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } } peers { permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 } }
I20260928 23:19:20.011768 13627 leader_election.cc:288] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [CANDIDATE]: Term 1 election: Requested vote from peers 8f487700fd534a1aa288ba5d8db77c16 (127.0.98.2:44555), 8fb6096151204d5f892f9b42b27fb1c6 (127.0.98.1:37727)
I20260928 23:19:20.011963 13272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1af877d8b70480a8bf7f4c130fb27fc" candidate_uuid: "47e480289bc2458ca164ac2153e692f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb6096151204d5f892f9b42b27fb1c6"
I20260928 23:19:20.012064 13272 raft_consensus.cc:3037] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:20.012943 13272 raft_consensus.cc:2454] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e480289bc2458ca164ac2153e692f4 in term 1.
I20260928 23:19:20.013197 13474 leader_election.cc:302] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47e480289bc2458ca164ac2153e692f4, 8fb6096151204d5f892f9b42b27fb1c6; no voters:
I20260928 23:19:20.013279 13627 raft_consensus.cc:2781] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:19:20.013417 13627 raft_consensus.cc:686] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 1 LEADER]: Becoming Leader. State: Replica: 47e480289bc2458ca164ac2153e692f4, State: Running, Role: LEADER
I20260928 23:19:20.013522 13627 consensus_queue.cc:237] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 } } peers { permanent_uuid: "47e480289bc2458ca164ac2153e692f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } } peers { permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 } }
I20260928 23:19:20.013674 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.999174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58307 (local address 127.0.98.2:44555)
0928 23:19:19.999230 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:19.999232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:20.010032 (+ 10800us) server_negotiation.cc:407] Connection header received
0928 23:19:20.010119 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:20.010120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:20.010132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:20.010158 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:20.010450 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:20.011080 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:20.012638 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:20.013210 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:20.013254 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:20.013264 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:20.013564 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 23:19:20.013609 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:19:20.014000 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:19.998994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44555 (local address 127.0.98.3:58307)
0928 23:19:20.009982 (+ 10988us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:20.009993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:20.010059 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:20.010284 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:20.010344 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:20.010346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:20.010381 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:20.010383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:20.011285 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:20.011286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:20.012545 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:20.012548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:20.013382 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:20.013382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:20.013440 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:20.013443 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:19:20.013896 (+ 453us) client_negotiation.cc:240] Negotiation successful
0928 23:19:20.013938 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10974,"thread_start_us":76,"threads_started":1}
I20260928 23:19:20.014308 13120 catalog_manager.cc:5495] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47e480289bc2458ca164ac2153e692f4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "47e480289bc2458ca164ac2153e692f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47e480289bc2458ca164ac2153e692f4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 } health_report { overall_health: UNKNOWN } } }
I20260928 23:19:20.014415 13406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1af877d8b70480a8bf7f4c130fb27fc" candidate_uuid: "47e480289bc2458ca164ac2153e692f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f487700fd534a1aa288ba5d8db77c16" is_pre_election: true
I20260928 23:19:20.014542 13406 raft_consensus.cc:2454] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e480289bc2458ca164ac2153e692f4 in term 0.
I20260928 23:19:20.014665 13405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1af877d8b70480a8bf7f4c130fb27fc" candidate_uuid: "47e480289bc2458ca164ac2153e692f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f487700fd534a1aa288ba5d8db77c16"
I20260928 23:19:20.014729 13405 raft_consensus.cc:3037] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:20.015520 13405 raft_consensus.cc:2454] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e480289bc2458ca164ac2153e692f4 in term 1.
I20260928 23:19:20.023502 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:20.020458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.98.3:37587)
0928 23:19:20.020493 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:20.020495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:20.020500 (+ 5us) server_negotiation.cc:407] Connection header received
0928 23:19:20.020528 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:20.020529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:20.020547 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:20.020602 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:19:20.020763 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:20.021461 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:20.022511 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:20.022786 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:20.022833 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:20.023018 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:19:20.023139 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:19:20.023201 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:20.023398 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 23:19:20.023416 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:19:20.025408 13272 raft_consensus.cc:1261] T d1af877d8b70480a8bf7f4c130fb27fc P 8fb6096151204d5f892f9b42b27fb1c6 [term 1 FOLLOWER]: Refusing update from remote peer 47e480289bc2458ca164ac2153e692f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:19:20.025763 13632 consensus_queue.cc:1035] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fb6096151204d5f892f9b42b27fb1c6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:19:20.027390 13405 raft_consensus.cc:1261] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16 [term 1 FOLLOWER]: Refusing update from remote peer 47e480289bc2458ca164ac2153e692f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:19:20.027710 13632 consensus_queue.cc:1035] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f487700fd534a1aa288ba5d8db77c16" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:19:20.032764 13635 mvcc.cc:205] Tried to move back new op lower bound from 7334451445861576704 to 7334451445815853056. Current Snapshot: MvccSnapshot[applied={T|T < 7334451445861576704}]
I20260928 23:19:20.034013 13638 mvcc.cc:205] Tried to move back new op lower bound from 7334451445861576704 to 7334451445815853056. Current Snapshot: MvccSnapshot[applied={T|T < 7334451445861576704}]
I20260928 23:19:20.036684 13120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:19:20.036775 13120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:19:20.037447 13120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P faf8b5bc96984e6d8bbc092920da58a8: Sending DeleteTablet for 3 replicas of tablet d1af877d8b70480a8bf7f4c130fb27fc
I20260928 23:19:20.038009 13386 tablet_service.cc:1514] Processing DeleteTablet for tablet d1af877d8b70480a8bf7f4c130fb27fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36680
I20260928 23:19:20.038271 13504 tablet_service.cc:1514] Processing DeleteTablet for tablet d1af877d8b70480a8bf7f4c130fb27fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40066
I20260928 23:19:20.038465 13252 tablet_service.cc:1514] Processing DeleteTablet for tablet d1af877d8b70480a8bf7f4c130fb27fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53658
I20260928 23:19:20.038969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 13158
I20260928 23:19:20.044375 13643 tablet_replica.cc:330] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: stopping tablet replica
I20260928 23:19:20.044529 13644 tablet_replica.cc:330] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: stopping tablet replica
I20260928 23:19:20.044937 13644 raft_consensus.cc:2229] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:19:20.045090 13643 raft_consensus.cc:2229] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:19:20.045231 13644 raft_consensus.cc:2258] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:19:20.047622 13643 raft_consensus.cc:2258] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:19:20.056066 13643 ts_tablet_manager.cc:1905] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:19:20.057917 13643 ts_tablet_manager.cc:1918] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 23:19:20.059046 13474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37727 (error 108)
I20260928 23:19:20.059366 13643 log.cc:1196] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/d1af877d8b70480a8bf7f4c130fb27fc
I20260928 23:19:20.059846 13643 ts_tablet_manager.cc:1939] T d1af877d8b70480a8bf7f4c130fb27fc P 8f487700fd534a1aa288ba5d8db77c16: Deleting consensus metadata
I20260928 23:19:20.059976 13644 ts_tablet_manager.cc:1905] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:19:20.060443 13107 catalog_manager.cc:4841] TS 8f487700fd534a1aa288ba5d8db77c16 (127.0.98.2:44555): tablet d1af877d8b70480a8bf7f4c130fb27fc (table test-table [id=6730d83243ed4115a1df37c50b8ce403]) successfully deleted
W20260928 23:19:20.060827 13105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37727 (error 108)
I20260928 23:19:20.061462 13644 ts_tablet_manager.cc:1918] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:19:20.061542 13644 log.cc:1196] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/d1af877d8b70480a8bf7f4c130fb27fc
I20260928 23:19:20.061676 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:20.061527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37727 (local address 127.0.98.3:40873)
0928 23:19:20.061605 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:20.061642 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 23:19:20.061774 13644 ts_tablet_manager.cc:1939] T d1af877d8b70480a8bf7f4c130fb27fc P 47e480289bc2458ca164ac2153e692f4: Deleting consensus metadata
I20260928 23:19:20.062175 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 13322
I20260928 23:19:20.062191 13106 catalog_manager.cc:4841] TS 47e480289bc2458ca164ac2153e692f4 (127.0.98.3:37587): tablet d1af877d8b70480a8bf7f4c130fb27fc (table test-table [id=6730d83243ed4115a1df37c50b8ce403]) successfully deleted
I20260928 23:19:20.063721 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:20.063488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37727 (local address 127.0.0.1:53666)
0928 23:19:20.063531 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:20.063692 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:19:20.063910 13105 catalog_manager.cc:4584] TS 8fb6096151204d5f892f9b42b27fb1c6 (127.0.98.1:37727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1af877d8b70480a8bf7f4c130fb27fc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37727: connect: Connection refused (error 111)
I20260928 23:19:20.067775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 13455
I20260928 23:19:20.074810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 13087
I20260928 23:19:20.081071 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 23:19:20.414148 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 23:19:20.414225 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 23:19:20.747876 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 23:19:20.748 UTC [12154] LOG: received smart shutdown request
2026-09-28 23:19:20.751 UTC [12154] LOG: background worker "logical replication launcher" (PID 12163) exited with exit code 1
2026-09-28 23:19:20.751 UTC [12158] LOG: shutting down
2026-09-28 23:19:20.893 UTC [12154] LOG: database system is shut down
2026-09-28T23:19:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44987 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:19:21 dist-test-slave-djkr krb5kdc[13657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:19:21 dist-test-slave-djkr krb5kdc[13657](info): set up 2 sockets
Sep 28 23:19:21 dist-test-slave-djkr krb5kdc[13657](info): commencing operation
krb5kdc: starting...
W20260928 23:19:21.927151 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:19:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:19:22.033887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46803
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46803,127.0.98.61:37477,127.0.98.60:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:22.109896 13670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:22.110100 13670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:22.110128 13670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:22.111210 13670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:19:22.111248 13670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:22.111265 13670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:22.111280 13670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:19:22.111295 13670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:19:22.112737 13670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46803,127.0.98.61:37477,127.0.98.60:45861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:22.112969 13670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:22.113157 13670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:22.117532 13675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:22.117527 13676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:22.117589 13670 server_base.cc:1034] running on GCE node
W20260928 23:19:22.117560 13678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:22.117904 13670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:22.118136 13670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:22.119272 13670 hybrid_clock.cc:648] HybridClock initialized: now 1790637562119260 us; error 26 us; skew 500 ppm
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:22.124864 13670 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:19:22.125257 13670 webserver.cc:460] Webserver started at http://127.0.98.62:46283/ using document root <none> and password file <none>
I20260928 23:19:22.125433 13670 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:22.125476 13670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:22.125592 13670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:22.126545 13670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "f40b6115b89f4b3e860dd3f0c228d2a2"
format_stamp: "Formatted at 2026-09-28 23:19:22 on dist-test-slave-djkr"
server_key: "53360dee9b340504e1ae7970e51053f7"
server_key_iv: "ef8935da49c88e783c79de3f8ac56ce8"
server_key_version: "encryptionkey@0"
I20260928 23:19:22.126856 13670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "f40b6115b89f4b3e860dd3f0c228d2a2"
format_stamp: "Formatted at 2026-09-28 23:19:22 on dist-test-slave-djkr"
server_key: "53360dee9b340504e1ae7970e51053f7"
server_key_iv: "ef8935da49c88e783c79de3f8ac56ce8"
server_key_version: "encryptionkey@0"
I20260928 23:19:22.128059 13670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:19:22.128669 13685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:22.128826 13670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:19:22.128901 13670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "f40b6115b89f4b3e860dd3f0c228d2a2"
format_stamp: "Formatted at 2026-09-28 23:19:22 on dist-test-slave-djkr"
server_key: "53360dee9b340504e1ae7970e51053f7"
server_key_iv: "ef8935da49c88e783c79de3f8ac56ce8"
server_key_version: "encryptionkey@0"
I20260928 23:19:22.128957 13670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:22.139938 13670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:22.141764 13670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:22.141882 13670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:22.145331 13670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46803
I20260928 23:19:22.145365 13737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46803 every 8 connection(s)
I20260928 23:19:22.145644 13670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:19:22.146379 13738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:22.147640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 13670
I20260928 23:19:22.147739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:19:22.147938 392 external_mini_cluster.cc:1428] Setting key 791c27c4b11e2f2ecb84535acf3a79dd
I20260928 23:19:22.147949 13738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260928 23:19:22.157388 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.148620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46803 (local address 127.0.0.1:55068)
0928 23:19:22.148860 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.148893 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:22.149410 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:22.149674 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:22.149676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:22.149887 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:22.150031 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.150034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.151362 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.151363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.152068 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.152070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.152694 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.152695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.152744 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.153154 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:22.153161 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:22.154396 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:22.156188 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.156194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.156196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.156366 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.156543 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.156543 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.156544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.156611 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.156873 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:22.156876 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:22.156940 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 23:19:22.157020 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:19:22.157233 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":100,"thread_start_us":61,"threads_started":1}
I20260928 23:19:22.157388 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.148634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.0.98.62:46803)
0928 23:19:22.148797 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.148805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.148963 (+ 158us) server_negotiation.cc:407] Connection header received
0928 23:19:22.149557 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.149559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.149579 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.149673 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.150110 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.151276 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.152143 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.152618 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.152725 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.154479 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.154486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.154487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.154493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.156112 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.156419 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.156421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.156422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.156489 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.156651 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.156652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.156652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.156829 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.156882 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.157046 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.157174 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260928 23:19:22.158179 13738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:19:22.158483 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.158374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37477 (local address 127.0.0.1:48728)
0928 23:19:22.158410 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.158448 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 23:19:22.158582 13687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37477: connect: Connection refused (error 111)
I20260928 23:19:22.158919 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.158835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37477 (local address 127.0.0.1:48740)
0928 23:19:22.158875 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.158898 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 23:19:22.159008 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37477: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37477: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 23:19:22.170198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37477
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46803,127.0.98.61:37477,127.0.98.60:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:19:22.208217 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } attempt: 1
I20260928 23:19:22.208745 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.208585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37477 (local address 127.0.0.1:48744)
0928 23:19:22.208652 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.208705 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 23:19:22.209111 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.209015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37477 (local address 127.0.0.1:48756)
0928 23:19:22.209062 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.209088 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 23:19:22.209208 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37477: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37477: connect: Connection refused (error 111)
W20260928 23:19:22.249485 13745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:22.249679 13745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:22.249707 13745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:22.251135 13745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:19:22.251179 13745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:22.251199 13745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:22.251215 13745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:19:22.251231 13745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:19:22.253377 13745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46803,127.0.98.61:37477,127.0.98.60:45861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:22.253656 13745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:22.253904 13745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:22.259248 13750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:22.259359 13751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:22.259250 13753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:22.259616 13745 server_base.cc:1034] running on GCE node
I20260928 23:19:22.259867 13745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:22.260198 13745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:22.261391 13745 hybrid_clock.cc:648] HybridClock initialized: now 1790637562261377 us; error 26 us; skew 500 ppm
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:22.264519 13745 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 23:19:22.265004 13745 webserver.cc:460] Webserver started at http://127.0.98.61:43637/ using document root <none> and password file <none>
I20260928 23:19:22.265256 13745 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:22.265312 13745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:22.265458 13745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:22.266774 13745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/data/instance:
uuid: "5d9264adade54d6f96a619a7dc0300de"
format_stamp: "Formatted at 2026-09-28 23:19:22 on dist-test-slave-djkr"
server_key: "6254f15ed50f0c073f46617913ce1b1a"
server_key_iv: "287f1a2f97305d2f5634e3f815b33811"
server_key_version: "encryptionkey@0"
I20260928 23:19:22.267124 13745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/wal/instance:
uuid: "5d9264adade54d6f96a619a7dc0300de"
format_stamp: "Formatted at 2026-09-28 23:19:22 on dist-test-slave-djkr"
server_key: "6254f15ed50f0c073f46617913ce1b1a"
server_key_iv: "287f1a2f97305d2f5634e3f815b33811"
server_key_version: "encryptionkey@0"
I20260928 23:19:22.268543 13745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:19:22.269383 13760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:22.269560 13745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:19:22.269654 13745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/wal
uuid: "5d9264adade54d6f96a619a7dc0300de"
format_stamp: "Formatted at 2026-09-28 23:19:22 on dist-test-slave-djkr"
server_key: "6254f15ed50f0c073f46617913ce1b1a"
server_key_iv: "287f1a2f97305d2f5634e3f815b33811"
server_key_version: "encryptionkey@0"
I20260928 23:19:22.269735 13745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:22.277344 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } attempt: 2
I20260928 23:19:22.277700 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.277591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37477 (local address 127.0.0.1:48760)
0928 23:19:22.277640 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.277672 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:19:22.278023 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.277933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37477 (local address 127.0.0.1:48762)
0928 23:19:22.277973 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.277997 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 23:19:22.278100 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37477: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37477: connect: Connection refused (error 111)
I20260928 23:19:22.286193 13745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:22.287941 13745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:22.288058 13745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:22.291715 13745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37477
I20260928 23:19:22.291805 13812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37477 every 8 connection(s)
I20260928 23:19:22.292145 13745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
I20260928 23:19:22.293004 13813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:22.294009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 13745
I20260928 23:19:22.294090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-1/wal/instance
I20260928 23:19:22.294265 392 external_mini_cluster.cc:1428] Setting key 487edb74ff25262d156c4b5339e43130
I20260928 23:19:22.294623 13813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 23:19:22.303031 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.295431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.0.98.62:46803)
0928 23:19:22.295476 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.295478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.295583 (+ 105us) server_negotiation.cc:407] Connection header received
0928 23:19:22.296094 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.296095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.296111 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.296157 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.296789 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.297313 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.298343 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.298555 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.298608 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.300611 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.300613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.300613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.300621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.301762 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.302102 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.302103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.302104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.302156 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.302357 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.302358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.302358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.302467 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.302553 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.302809 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.302890 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260928 23:19:22.303135 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.295257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46803 (local address 127.0.0.1:55082)
0928 23:19:22.295442 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.295492 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:22.295973 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:22.296259 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:22.296262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:22.296509 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:22.296703 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.296708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.297407 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.297408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.298259 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.298262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.298621 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.298621 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.298740 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.299184 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:22.299192 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:22.300524 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:22.301831 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.301838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.301840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.302040 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.302207 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.302208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.302208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.302305 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.302547 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:22.302553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:22.302695 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 23:19:22.302783 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:19:22.302923 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260928 23:19:22.303653 13813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:19:22.310770 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.303877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37477 (local address 127.0.0.1:48764)
0928 23:19:22.303918 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.303930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:22.303969 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:22.304185 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:22.304186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:22.304316 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:22.304372 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.304374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.305565 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.305566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.306288 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.306290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.306757 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.306758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.306834 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.307161 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:22.307167 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:22.308440 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:22.309911 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.309912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.309912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.310105 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.310272 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.310272 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.310273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.310334 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.310544 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:22.310544 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:22.310575 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 23:19:22.310619 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:19:22.310690 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 23:19:22.310842 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.303935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.0.98.61:37477)
0928 23:19:22.304064 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.304068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.304086 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:19:22.304106 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.304107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.304118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.304173 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.304447 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.305497 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.306356 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.306688 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.306740 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.308514 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.308521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.308521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.308526 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.309834 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.310172 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.310174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.310174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.310220 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.310380 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.310380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.310381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.310500 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.310553 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.310641 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.310733 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260928 23:19:22.311394 13813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:19:22.311733 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.311628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56442)
0928 23:19:22.311661 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.311711 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 23:19:22.311842 13763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
I20260928 23:19:22.312214 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.312128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56458)
0928 23:19:22.312169 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.312193 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 23:19:22.312304 13813 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45861: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 23:19:22.317955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45861
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46803,127.0.98.61:37477,127.0.98.60:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:19:22.361492 13813 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } attempt: 1
I20260928 23:19:22.361965 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.361845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56466)
0928 23:19:22.361901 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.361939 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 23:19:22.362259 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.362185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56470)
0928 23:19:22.362220 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.362242 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 23:19:22.362329 13813 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45861: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
I20260928 23:19:22.369218 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } attempt: 3
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:19:22.375540 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.369566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.0.98.61:37477)
0928 23:19:22.369618 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.369620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.369630 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:19:22.369675 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.369675 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.369688 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.369735 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.370042 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.370546 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.371223 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.371426 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.371486 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.373148 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.373150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.373150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.373157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.374502 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.374860 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.374861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.374862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.374915 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.375077 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.375077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.375078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.375186 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.375239 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.375370 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.375441 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:19:22.375540 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.369514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37477 (local address 127.0.0.1:48770)
0928 23:19:22.369562 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.369573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:22.369620 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:22.369746 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:22.369747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:22.369893 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:22.369955 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.369957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.370617 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.370618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.371155 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.371157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.371488 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.371489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.371588 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.371978 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:22.371983 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:22.373073 (+ 1090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:22.374622 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.374623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.374624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.374803 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.374971 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.374972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.374972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.375028 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.375237 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:22.375238 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:22.375286 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 23:19:22.375343 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:19:22.375441 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 23:19:22.375871 13738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:19:22.376169 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.376084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56480)
0928 23:19:22.376122 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.376147 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 23:19:22.376464 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.376394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56488)
0928 23:19:22.376431 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.376451 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:19:22.376549 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45861: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
W20260928 23:19:22.395758 13820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:22.395952 13820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:22.395972 13820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:22.397043 13820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:19:22.397073 13820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:22.397084 13820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:22.397095 13820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:19:22.397112 13820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:19:22.398609 13820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46803,127.0.98.61:37477,127.0.98.60:45861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:22.398799 13820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:22.398972 13820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:22.403299 13826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:22.403411 13820 server_base.cc:1034] running on GCE node
W20260928 23:19:22.403287 13828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:22.403287 13825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:22.403725 13820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:22.404013 13820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:22.405149 13820 hybrid_clock.cc:648] HybridClock initialized: now 1790637562405138 us; error 28 us; skew 500 ppm
I20260928 23:19:22.407680 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } attempt: 1
I20260928 23:19:22.408093 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.407964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56494)
0928 23:19:22.408019 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.408061 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 23:19:22.408746 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.408654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56510)
0928 23:19:22.408691 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.408720 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 23:19:22.409655 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45861: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45861: connect: Connection refused (error 111)
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:22.410661 13820 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 23:19:22.411092 13820 webserver.cc:460] Webserver started at http://127.0.98.60:37245/ using document root <none> and password file <none>
I20260928 23:19:22.411291 13820 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:22.411335 13820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:22.411456 13820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:22.412485 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/data/instance:
uuid: "4281a3c5d01342ae8003d4c317f63f94"
format_stamp: "Formatted at 2026-09-28 23:19:22 on dist-test-slave-djkr"
server_key: "b0bf37d933e7e0215b19a0ea1a6a189d"
server_key_iv: "1b562829f8aba953519d435349377dac"
server_key_version: "encryptionkey@0"
I20260928 23:19:22.412801 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/wal/instance:
uuid: "4281a3c5d01342ae8003d4c317f63f94"
format_stamp: "Formatted at 2026-09-28 23:19:22 on dist-test-slave-djkr"
server_key: "b0bf37d933e7e0215b19a0ea1a6a189d"
server_key_iv: "1b562829f8aba953519d435349377dac"
server_key_version: "encryptionkey@0"
I20260928 23:19:22.413946 13820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:19:22.414539 13835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:22.414675 13820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:19:22.414749 13820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/wal
uuid: "4281a3c5d01342ae8003d4c317f63f94"
format_stamp: "Formatted at 2026-09-28 23:19:22 on dist-test-slave-djkr"
server_key: "b0bf37d933e7e0215b19a0ea1a6a189d"
server_key_iv: "1b562829f8aba953519d435349377dac"
server_key_version: "encryptionkey@0"
I20260928 23:19:22.414812 13820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:22.424088 13820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:22.425621 13820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:22.425738 13820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:22.429216 13820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45861
I20260928 23:19:22.429243 13887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45861 every 8 connection(s)
I20260928 23:19:22.429525 13820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
I20260928 23:19:22.430231 13888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:22.430441 13813 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } attempt: 2
I20260928 23:19:22.431721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 13820
I20260928 23:19:22.431813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637417635540-392-0/minicluster-data/master-2/wal/instance
I20260928 23:19:22.432001 392 external_mini_cluster.cc:1428] Setting key 9a951df319cdca0b71338ac0304032b7
I20260928 23:19:22.432257 13888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:19:22.439311 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.430680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56522)
0928 23:19:22.430733 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.430744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:22.430785 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:22.431744 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:22.431745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:22.431918 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:22.431968 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.431970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.433338 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.433339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.433949 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.433952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.434562 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.434563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.434625 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.435026 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:22.435031 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:22.436416 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:22.438244 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.438245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.438245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.438403 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.438629 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.438630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.438630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.438694 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.439012 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:22.439013 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:22.439057 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:19:22.439125 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:19:22.439215 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 23:19:22.439795 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.430799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.0.98.60:45861)
0928 23:19:22.430998 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.431005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.431028 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:19:22.431471 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.431477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.431522 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.431708 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.432063 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.433258 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.434025 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.434449 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.434588 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.436499 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.436506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.436507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.436513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.438172 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.438481 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.438484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.438485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.438573 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.438739 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.438740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.438740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.438945 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.439017 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.439209 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.439324 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
I20260928 23:19:22.440328 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.432872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.0.98.62:46803)
0928 23:19:22.432918 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.432920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.433085 (+ 165us) server_negotiation.cc:407] Connection header received
0928 23:19:22.433134 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.433135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.433146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.433195 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.433633 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.434354 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.435216 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.435496 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.435571 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.437724 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.437725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.437725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.437732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.439080 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.439492 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.439493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.439493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.439585 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.439779 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.439780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.439780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.439902 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.439962 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.440168 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.440246 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:19:22.440584 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.432824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46803 (local address 127.0.0.1:55094)
0928 23:19:22.432980 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.433018 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:22.433082 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:22.433212 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:22.433214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:22.433505 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:22.433556 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.433558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.434440 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.434441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.435144 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.435146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.435601 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.435602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.435681 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.436067 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:22.436076 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:22.437600 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:22.439158 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.439162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.439164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.439404 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.439657 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.439658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.439658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.439716 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.440000 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:22.440004 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:22.440047 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 23:19:22.440389 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 23:19:22.440492 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260928 23:19:22.441309 13813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de: Bootstrap starting.
I20260928 23:19:22.441751 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.432292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.0.98.60:45861)
0928 23:19:22.432719 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.432721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.432848 (+ 127us) server_negotiation.cc:407] Connection header received
0928 23:19:22.432947 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.432948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.432956 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.432996 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.433703 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.434432 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.435318 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.435615 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.435669 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.438045 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.438047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.438047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.438053 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.440810 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.441086 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.441087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.441087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.441133 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.441282 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.441283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.441283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.441388 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.441499 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.441582 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.441647 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":72,"threads_started":1}
I20260928 23:19:22.442174 13813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:22.442482 13813 log.cc:824] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:22.443038 13888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:19:22.443318 13813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de: No bootstrap required, opened a new log
I20260928 23:19:22.445258 13813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } }
I20260928 23:19:22.445389 13813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:22.445417 13813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9264adade54d6f96a619a7dc0300de, State: Initialized, Role: FOLLOWER
I20260928 23:19:22.445549 13813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } }
I20260928 23:19:22.445845 13813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:22.445807 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } } }
I20260928 23:19:22.446017 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260928 23:19:22.447640 13906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:19:22.447687 13906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:19:22.449621 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.442300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.0.98.62:46803)
0928 23:19:22.442331 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.442333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.442343 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:19:22.442405 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.442406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.442413 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.442451 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.442953 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.443529 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.444159 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.444352 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.444401 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.446361 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.446362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.446363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.446369 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.447908 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.448246 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.448246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.448247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.448296 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.448480 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.448481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.448481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.448596 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.449130 (+ 534us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.449461 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.449539 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:19:22.450169 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.443611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.0.98.61:37477)
0928 23:19:22.443646 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.443648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.443655 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:19:22.443671 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.443672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.443679 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.443725 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.444017 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.444670 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.445559 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.445830 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.445871 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.447730 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.447731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.447732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.447738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.449145 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.449481 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.449482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.449483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.449532 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.449721 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.449721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.449722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.449838 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.449891 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.450030 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.450091 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:19:22.450166 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.443251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37477 (local address 127.0.0.1:48774)
0928 23:19:22.443286 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.443294 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:22.443325 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:22.443763 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:22.443764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:22.443891 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:22.443936 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.443938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.444775 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.444776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.445462 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.445474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.445941 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.445942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.446006 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.446345 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:22.446350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:22.447637 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:22.449223 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.449224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.449225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.449405 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.449597 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.449597 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.449598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.449648 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.449897 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:22.449898 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:22.449938 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 23:19:22.450002 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:19:22.450080 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:19:22.448266 13745 master_runner.cc:186] Error getting master registration for 127.0.98.62:46803: OK, instance_id { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" instance_seqno: 1790637562144394 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:19:22.451007 13745 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:19:22.451666 13888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:19:22.458976 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.451806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.0.98.61:37477)
0928 23:19:22.451870 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.451871 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.451879 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:19:22.451919 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.451920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.451928 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.451979 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.452498 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.453124 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.453699 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.453917 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.453967 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.456654 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.456655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.456655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.456662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.458012 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.458332 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.458333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.458333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.458382 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.458558 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.458558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.458559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.458669 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.458807 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.458840 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.458899 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:19:22.459251 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.451870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56540)
0928 23:19:22.451975 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.451983 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:22.452015 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:22.452121 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:22.452122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:22.452233 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:22.452253 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.452254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.453039 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.453040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.453782 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.453784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.454233 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.454234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.454266 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.454587 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:22.454591 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:22.456575 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:22.458265 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.458265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.458266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.458479 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.458657 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.458657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.458658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.458716 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.458999 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:22.458999 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:22.459031 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 23:19:22.459083 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:19:22.459163 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 23:19:22.459478 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.451917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.0.98.60:45861)
0928 23:19:22.451951 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.451952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.452052 (+ 100us) server_negotiation.cc:407] Connection header received
0928 23:19:22.452065 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.452065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.452071 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.452106 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.452310 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.452942 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.453864 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.454141 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.454196 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.456840 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.456841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.456841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.456846 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.458188 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.458551 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.458552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.458552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.458599 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.458778 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.458779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.458779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.458925 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.458979 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.459343 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.459404 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:19:22.460505 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:19:22.460847 13888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94: Bootstrap starting.
I20260928 23:19:22.461540 13888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:22.461827 13888 log.cc:824] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:22.462656 13888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94: No bootstrap required, opened a new log
I20260928 23:19:22.464622 13888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } }
I20260928 23:19:22.464776 13888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:22.464809 13888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4281a3c5d01342ae8003d4c317f63f94, State: Initialized, Role: FOLLOWER
I20260928 23:19:22.464912 13888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } }
I20260928 23:19:22.465173 13888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:19:22.469054 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.461890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.0.98.62:46803)
0928 23:19:22.462105 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.462107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.462298 (+ 191us) server_negotiation.cc:407] Connection header received
0928 23:19:22.462386 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.462387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.462397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.462424 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.463052 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.463735 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.464701 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.464991 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.465035 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.466495 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.466496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.466497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.466502 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.467928 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.468280 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.468281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.468281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.468330 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.468507 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.468507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.468508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.468620 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.468695 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.468882 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.468956 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260928 23:19:22.466926 13820 master_runner.cc:186] Error getting master registration for 127.0.98.62:46803: OK, instance_id { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" instance_seqno: 1790637562144394 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:19:22.469563 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.462013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.0.98.60:45861)
0928 23:19:22.462046 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.462047 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.462177 (+ 130us) server_negotiation.cc:407] Connection header received
0928 23:19:22.462221 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.462221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.462230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.462281 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.462909 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.463663 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.464671 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.464974 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.465011 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.466560 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.466561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.466561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.466568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.468453 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.468804 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.468805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.468805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.468852 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.469034 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.469035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.469035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.469142 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.469214 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.469401 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.469481 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:19:22.469568 13820 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 23:19:22.470229 13926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:19:22.470275 13926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:19:22.470506 13915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } } }
I20260928 23:19:22.470561 13915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:19:22.475730 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.470772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.0.98.61:37477)
0928 23:19:22.470816 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.470817 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.470825 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:19:22.470855 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.470855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.470863 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.470902 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.471298 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.471783 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.472358 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.472536 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.472585 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.473552 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.473553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.473554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.473560 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.474776 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.475095 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.475096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.475097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.475146 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.475292 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.475292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.475293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.475389 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.475461 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.475583 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.475649 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:19:22.484799 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } attempt: 2
Sep 28 23:19:22 dist-test-slave-djkr krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:19:22.490630 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.485063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.0.98.60:45861)
0928 23:19:22.485094 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:22.485096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:22.485103 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:19:22.485152 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:22.485152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:22.485161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:22.485203 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:22.485423 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.485868 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.486506 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.486678 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.486727 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.488581 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:22.488582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:22.488583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:22.488588 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:22.489720 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.489997 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.489998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.489999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.490044 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:22.490211 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:22.490212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:22.490212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:22.490303 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:22.490353 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:22.490480 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 23:19:22.490545 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:19:22.490635 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:22.485012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45861 (local address 127.0.0.1:56548)
0928 23:19:22.485053 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:22.485062 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:22.485097 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:22.485211 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:22.485212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:22.485331 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:22.485365 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.485366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.485935 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.485936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.486441 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:22.486442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:22.486731 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:22.486732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:22.486781 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:22.487123 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:22.487127 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:22.488513 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:22.489790 (+ 1277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.489791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.489791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.489939 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.490109 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:22.490109 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:22.490110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:22.490163 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:22.490354 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:22.490355 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:22.490396 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:19:22.490455 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:19:22.490545 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 23:19:22.491991 13738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2: Bootstrap starting.
I20260928 23:19:22.492556 13738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:22.492856 13738 log.cc:824] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:22.493744 13738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2: No bootstrap required, opened a new log
I20260928 23:19:22.495113 13738 raft_consensus.cc:348] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } }
I20260928 23:19:22.495217 13738 raft_consensus.cc:374] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:22.495247 13738 raft_consensus.cc:729] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40b6115b89f4b3e860dd3f0c228d2a2, State: Initialized, Role: FOLLOWER
I20260928 23:19:22.495371 13738 consensus_queue.cc:260] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } }
I20260928 23:19:22.495699 13738 sys_catalog.cc:564] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 23:19:22.497220 13940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:19:22.497309 13940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:19:22.495663 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } } }
I20260928 23:19:22.498621 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:19:22.499281 13670 master_runner.cc:263] No leader master found from master f40b6115b89f4b3e860dd3f0c228d2a2
I20260928 23:19:22.499328 13670 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 23:19:22.538596 13766 tablet.cc:2369] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:19:22.595468 13895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:19:22.595621 13895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } }
I20260928 23:19:22.595857 13895 leader_election.cc:288] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f40b6115b89f4b3e860dd3f0c228d2a2 (127.0.98.62:46803), 4281a3c5d01342ae8003d4c317f63f94 (127.0.98.60:45861)
I20260928 23:19:22.596253 13712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d9264adade54d6f96a619a7dc0300de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" is_pre_election: true
I20260928 23:19:22.596333 13862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d9264adade54d6f96a619a7dc0300de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4281a3c5d01342ae8003d4c317f63f94" is_pre_election: true
I20260928 23:19:22.596428 13712 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9264adade54d6f96a619a7dc0300de in term 0.
I20260928 23:19:22.596510 13862 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9264adade54d6f96a619a7dc0300de in term 0.
I20260928 23:19:22.596742 13763 leader_election.cc:302] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4281a3c5d01342ae8003d4c317f63f94, 5d9264adade54d6f96a619a7dc0300de; no voters:
I20260928 23:19:22.597052 13895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:19:22.597116 13895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:19:22.597144 13895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:22.598207 13895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } }
I20260928 23:19:22.598335 13895 leader_election.cc:288] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [CANDIDATE]: Term 1 election: Requested vote from peers f40b6115b89f4b3e860dd3f0c228d2a2 (127.0.98.62:46803), 4281a3c5d01342ae8003d4c317f63f94 (127.0.98.60:45861)
I20260928 23:19:22.598520 13712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d9264adade54d6f96a619a7dc0300de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2"
I20260928 23:19:22.598609 13712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:22.598618 13862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d9264adade54d6f96a619a7dc0300de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4281a3c5d01342ae8003d4c317f63f94"
I20260928 23:19:22.598692 13862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:22.599524 13712 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9264adade54d6f96a619a7dc0300de in term 1.
I20260928 23:19:22.599592 13862 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9264adade54d6f96a619a7dc0300de in term 1.
I20260928 23:19:22.599766 13764 leader_election.cc:302] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9264adade54d6f96a619a7dc0300de, f40b6115b89f4b3e860dd3f0c228d2a2; no voters:
I20260928 23:19:22.599882 13895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:19:22.600070 13895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [term 1 LEADER]: Becoming Leader. State: Replica: 5d9264adade54d6f96a619a7dc0300de, State: Running, Role: LEADER
I20260928 23:19:22.600165 13895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } }
I20260928 23:19:22.600528 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9264adade54d6f96a619a7dc0300de. Latest consensus state: current_term: 1 leader_uuid: "5d9264adade54d6f96a619a7dc0300de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } } }
I20260928 23:19:22.600615 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:22.601221 13944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:19:22.601434 13944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:19:22.603434 13862 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [term 1 FOLLOWER]: Refusing update from remote peer 5d9264adade54d6f96a619a7dc0300de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:19:22.603752 13895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [LEADER]: Connected to new peer: Peer: permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:19:22.604197 13712 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [term 1 FOLLOWER]: Refusing update from remote peer 5d9264adade54d6f96a619a7dc0300de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:19:22.604480 13895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:19:22.605099 13915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9264adade54d6f96a619a7dc0300de. Latest consensus state: current_term: 1 leader_uuid: "5d9264adade54d6f96a619a7dc0300de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } } }
I20260928 23:19:22.605185 13915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:19:22.605623 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d9264adade54d6f96a619a7dc0300de. Latest consensus state: current_term: 1 leader_uuid: "5d9264adade54d6f96a619a7dc0300de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } } }
I20260928 23:19:22.605715 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:19:22.608741 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d9264adade54d6f96a619a7dc0300de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } } }
I20260928 23:19:22.608819 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:22.609014 13915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d9264adade54d6f96a619a7dc0300de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } } }
I20260928 23:19:22.609091 13915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:19:22.609324 13945 mvcc.cc:205] Tried to move back new op lower bound from 7334451456421052416 to 7334451456410767360. Current Snapshot: MvccSnapshot[applied={T|T < 7334451456421052416}]
I20260928 23:19:22.610423 13946 mvcc.cc:205] Tried to move back new op lower bound from 7334451456421052416 to 7334451456410767360. Current Snapshot: MvccSnapshot[applied={T|T < 7334451456421052416}]
I20260928 23:19:22.611821 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d9264adade54d6f96a619a7dc0300de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } } }
I20260928 23:19:22.611914 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:22.612329 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d9264adade54d6f96a619a7dc0300de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40b6115b89f4b3e860dd3f0c228d2a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46803 } } peers { permanent_uuid: "5d9264adade54d6f96a619a7dc0300de" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37477 } } peers { permanent_uuid: "4281a3c5d01342ae8003d4c317f63f94" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45861 } } }
I20260928 23:19:22.612416 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:19:22.612511 13948 mvcc.cc:205] Tried to move back new op lower bound from 7334451456421052416 to 7334451456410767360. Current Snapshot: MvccSnapshot[applied={T|T < 7334451456421052416}]
I20260928 23:19:22.613029 13944 catalog_manager.cc:1347] Generated new cluster ID: d2b07092c81444d1b7c5af2f524bdcb7
I20260928 23:19:22.613067 13944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:19:22.632833 13944 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:19:22.633345 13944 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:19:22.636983 13944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d9264adade54d6f96a619a7dc0300de: Generated new TSK 0
I20260928 23:19:22.637151 13944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:19:23.470734 13926 catalog_manager.cc:1259] Loaded cluster ID: d2b07092c81444d1b7c5af2f524bdcb7
I20260928 23:19:23.470808 13926 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94: loading cluster ID for follower catalog manager: success
I20260928 23:19:23.471591 13926 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94: acquiring CA information for follower catalog manager: success
I20260928 23:19:23.471881 13926 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4281a3c5d01342ae8003d4c317f63f94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:19:23.497757 13940 catalog_manager.cc:1259] Loaded cluster ID: d2b07092c81444d1b7c5af2f524bdcb7
I20260928 23:19:23.497834 13940 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2: loading cluster ID for follower catalog manager: success
I20260928 23:19:23.498627 13940 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2: acquiring CA information for follower catalog manager: success
I20260928 23:19:23.498869 13940 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f40b6115b89f4b3e860dd3f0c228d2a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:19:23.687747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 13670
I20260928 23:19:23.692305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 13745
W20260928 23:19:23.696617 13837 connection.cc:535] server connection from 127.0.0.1:56522 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:19:23.696763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 13820
2026-09-28T23:19:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2716 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:19:23 dist-test-slave-djkr krb5kdc[13959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:19:23 dist-test-slave-djkr krb5kdc[13959](info): set up 2 sockets
Sep 28 23:19:23 dist-test-slave-djkr krb5kdc[13959](info): commencing operation
krb5kdc: starting...
W20260928 23:19:24.639353 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:19:24 dist-test-slave-djkr krb5kdc[13959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:19:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:19:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:19:24.747658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44453
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:24.824257 13972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:24.824455 13972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:24.824482 13972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:24.825603 13972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:19:24.825642 13972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:24.825659 13972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:24.825675 13972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:19:24.825690 13972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:19:24.827093 13972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:24.827332 13972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:24.827533 13972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:24.832101 13977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:24.832146 13972 server_base.cc:1034] running on GCE node
W20260928 23:19:24.832116 13978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:24.832116 13980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:24.832527 13972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:24.832815 13972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:24.833956 13972 hybrid_clock.cc:648] HybridClock initialized: now 1790637564833942 us; error 30 us; skew 500 ppm
Sep 28 23:19:24 dist-test-slave-djkr krb5kdc[13959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:24.836365 13972 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:19:24.836807 13972 webserver.cc:460] Webserver started at http://127.0.98.62:46333/ using document root <none> and password file <none>
I20260928 23:19:24.837066 13972 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:24.837116 13972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:24.837234 13972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:24.838254 13972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "8e7f79bb4d6c476aac802dc1f61867ed"
format_stamp: "Formatted at 2026-09-28 23:19:24 on dist-test-slave-djkr"
server_key: "9ce415c32a2f63922c62a565a4027c5b"
server_key_iv: "c8619623e2a8a8e45cdb3f16fe4ac891"
server_key_version: "encryptionkey@0"
I20260928 23:19:24.838573 13972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e7f79bb4d6c476aac802dc1f61867ed"
format_stamp: "Formatted at 2026-09-28 23:19:24 on dist-test-slave-djkr"
server_key: "9ce415c32a2f63922c62a565a4027c5b"
server_key_iv: "c8619623e2a8a8e45cdb3f16fe4ac891"
server_key_version: "encryptionkey@0"
I20260928 23:19:24.839819 13972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:19:24.840533 13987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:24.840670 13972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:19:24.840754 13972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "8e7f79bb4d6c476aac802dc1f61867ed"
format_stamp: "Formatted at 2026-09-28 23:19:24 on dist-test-slave-djkr"
server_key: "9ce415c32a2f63922c62a565a4027c5b"
server_key_iv: "c8619623e2a8a8e45cdb3f16fe4ac891"
server_key_version: "encryptionkey@0"
I20260928 23:19:24.840812 13972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:24.857749 13972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:24.859449 13972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:24.859602 13972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:24.863252 13972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44453
I20260928 23:19:24.863283 14039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44453 every 8 connection(s)
I20260928 23:19:24.863652 13972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:19:24.864439 14040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:24.866556 14040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed: Bootstrap starting.
I20260928 23:19:24.867058 14040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:24.867268 14040 log.cc:824] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:24.867864 14040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed: No bootstrap required, opened a new log
I20260928 23:19:24.869323 14040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7f79bb4d6c476aac802dc1f61867ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44453 } }
I20260928 23:19:24.869451 14040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:24.869472 14040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7f79bb4d6c476aac802dc1f61867ed, State: Initialized, Role: FOLLOWER
I20260928 23:19:24.869570 14040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7f79bb4d6c476aac802dc1f61867ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44453 } }
I20260928 23:19:24.869614 14040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:19:24.869635 14040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:19:24.869664 14040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:24.870093 14040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7f79bb4d6c476aac802dc1f61867ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44453 } }
I20260928 23:19:24.870174 14040 leader_election.cc:302] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e7f79bb4d6c476aac802dc1f61867ed; no voters:
I20260928 23:19:24.870277 14040 leader_election.cc:288] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:19:24.870337 14043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:19:24.870487 14040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:19:24.870510 14043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [term 1 LEADER]: Becoming Leader. State: Replica: 8e7f79bb4d6c476aac802dc1f61867ed, State: Running, Role: LEADER
I20260928 23:19:24.870673 14043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7f79bb4d6c476aac802dc1f61867ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44453 } }
I20260928 23:19:24.871284 14045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e7f79bb4d6c476aac802dc1f61867ed. Latest consensus state: current_term: 1 leader_uuid: "8e7f79bb4d6c476aac802dc1f61867ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7f79bb4d6c476aac802dc1f61867ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44453 } } }
I20260928 23:19:24.871364 14045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:24.871747 14052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:19:24.871810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 13972
I20260928 23:19:24.871896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:19:24.872069 392 external_mini_cluster.cc:1428] Setting key b6ce3fe9000549b806488f4f8e285671
I20260928 23:19:24.871804 14044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e7f79bb4d6c476aac802dc1f61867ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e7f79bb4d6c476aac802dc1f61867ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44453 } } }
I20260928 23:19:24.872184 14052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:19:24.872383 14044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:24.873713 14052 catalog_manager.cc:1347] Generated new cluster ID: 4f68adf96438451ca1f2321da46d577e
I20260928 23:19:24.873759 14052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:19:24 dist-test-slave-djkr krb5kdc[13959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:24.878564 14052 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:19:24.879206 14052 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:19:24.881493 14062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:24.872643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.0.98.62:44453)
0928 23:19:24.872830 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:24.872840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:24.872909 (+ 69us) server_negotiation.cc:407] Connection header received
0928 23:19:24.873020 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:24.873028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:24.873083 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:24.873256 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:24.873895 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:24.874784 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:24.875490 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:24.875943 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:24.876045 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:24.878078 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:24.878082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:24.878082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:24.878093 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:24.880213 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:24.880525 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:24.880528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:24.880528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:24.880608 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:24.880769 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:24.880770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:24.880770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:24.880977 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:24.881125 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:24.881200 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 23:19:24.881337 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260928 23:19:24.882252 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:19:24.884956 14052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e7f79bb4d6c476aac802dc1f61867ed: Generated new TSK 0
I20260928 23:19:24.885104 14052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:19:24.935767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 13972
2026-09-28T23:19:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1238 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:19:24 dist-test-slave-djkr krb5kdc[14067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:19:24 dist-test-slave-djkr krb5kdc[14067](info): set up 2 sockets
Sep 28 23:19:24 dist-test-slave-djkr krb5kdc[14067](info): commencing operation
krb5kdc: starting...
W20260928 23:19:25.883273 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:19:25 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:19:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:19:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:19:25.990522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44711
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:26.068290 14080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:26.068486 14080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:26.068507 14080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:26.069624 14080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:19:26.069670 14080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:26.069684 14080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:26.069707 14080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:19:26.069731 14080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:19:26.071187 14080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:26.071403 14080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:26.071621 14080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:19:26.076278 14080 server_base.cc:1034] running on GCE node
W20260928 23:19:26.076278 14085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:26.076242 14088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:26.076234 14086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:26.076679 14080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:26.076931 14080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:26.078068 14080 hybrid_clock.cc:648] HybridClock initialized: now 1790637566078048 us; error 37 us; skew 500 ppm
Sep 28 23:19:26 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:26.080300 14080 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:19:26.080657 14080 webserver.cc:460] Webserver started at http://127.0.98.62:35989/ using document root <none> and password file <none>
I20260928 23:19:26.080822 14080 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:26.080859 14080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:26.080945 14080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:26.081887 14080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "898107bef00d473c842ff384f491c3b2"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "01225fa8211d30a90cbd0a7470ba0c08"
server_key_iv: "298a70ab06c5a7594bd861bbe5d1edba"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.082176 14080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "898107bef00d473c842ff384f491c3b2"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "01225fa8211d30a90cbd0a7470ba0c08"
server_key_iv: "298a70ab06c5a7594bd861bbe5d1edba"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.083273 14080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:19:26.083941 14095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.084095 14080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.084162 14080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "898107bef00d473c842ff384f491c3b2"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "01225fa8211d30a90cbd0a7470ba0c08"
server_key_iv: "298a70ab06c5a7594bd861bbe5d1edba"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.084225 14080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:26.099436 14080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:26.101377 14080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:26.101492 14080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:26.105165 14080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44711
I20260928 23:19:26.105195 14147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44711 every 8 connection(s)
I20260928 23:19:26.105506 14080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:19:26.106395 14148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:26.108690 14148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Bootstrap starting.
I20260928 23:19:26.109232 14148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:26.109452 14148 log.cc:824] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:26.110046 14148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: No bootstrap required, opened a new log
I20260928 23:19:26.111537 14148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:26.111716 14148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:26.111738 14148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898107bef00d473c842ff384f491c3b2, State: Initialized, Role: FOLLOWER
I20260928 23:19:26.111842 14148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:26.111891 14148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:19:26.111912 14148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:19:26.111941 14148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:26.112660 14148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:26.112746 14148 leader_election.cc:302] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 898107bef00d473c842ff384f491c3b2; no voters:
I20260928 23:19:26.112864 14148 leader_election.cc:288] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:19:26.112911 14151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:19:26.113070 14148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:19:26.113076 14151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 898107bef00d473c842ff384f491c3b2, State: Running, Role: LEADER
I20260928 23:19:26.113225 14151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:26.113570 14153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 898107bef00d473c842ff384f491c3b2. Latest consensus state: current_term: 1 leader_uuid: "898107bef00d473c842ff384f491c3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } } }
I20260928 23:19:26.113754 14153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:26.113699 14152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "898107bef00d473c842ff384f491c3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } } }
I20260928 23:19:26.113953 14152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:26.113955 14160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:19:26.114452 14160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:19:26.114627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 14080
I20260928 23:19:26.114724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:19:26.114898 392 external_mini_cluster.cc:1428] Setting key 2b0875820b371a832697205e5a902622
I20260928 23:19:26.115935 14160 catalog_manager.cc:1347] Generated new cluster ID: 2adeaec8dbb34980b3a4fbb59b0b927d
I20260928 23:19:26.115988 14160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:19:26 dist-test-slave-djkr krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:26.126461 14160 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:19:26.127125 14160 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:19:26.128108 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.115305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.0.98.62:44711)
0928 23:19:26.115464 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.115472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.115667 (+ 195us) server_negotiation.cc:407] Connection header received
0928 23:19:26.115867 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.115875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.119655 (+ 3780us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.119758 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:26.120439 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.121639 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.122360 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.122772 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.122865 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.125019 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:26.125023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:26.125023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:26.125036 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:26.126669 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:26.126976 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:26.126979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:26.126979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:26.127072 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:26.127238 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:26.127239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:26.127240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:26.127449 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:26.127599 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.127702 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.127932 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260928 23:19:26.139628 14160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Generated new TSK 0
I20260928 23:19:26.139802 14160 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:19:26.165853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44711
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:26.261065 14174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:26.261253 14174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:26.261291 14174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:26.262382 14174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:26.262421 14174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:26.262454 14174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:19:26.263850 14174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:26.264060 14174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:26.264251 14174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:26.268805 14179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:26.268815 14180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:26.268934 14182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:26.269002 14174 server_base.cc:1034] running on GCE node
I20260928 23:19:26.269217 14174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:26.269398 14174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:26.270529 14174 hybrid_clock.cc:648] HybridClock initialized: now 1790637566270520 us; error 26 us; skew 500 ppm
Sep 28 23:19:26 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:26.272754 14174 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:19:26.273147 14174 webserver.cc:460] Webserver started at http://127.0.98.1:38129/ using document root <none> and password file <none>
I20260928 23:19:26.273337 14174 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:26.273384 14174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:26.273501 14174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:26.274446 14174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "7c9e483323274aa1968278b79d1ec0f0"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "8c905cefe2513e76b01d6d603a0647d7"
server_key_iv: "95eb17f9bdaa152a56d5b165155232be"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.274755 14174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7c9e483323274aa1968278b79d1ec0f0"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "8c905cefe2513e76b01d6d603a0647d7"
server_key_iv: "95eb17f9bdaa152a56d5b165155232be"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.275935 14174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:19:26.276535 14189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.276679 14174 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.276753 14174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "7c9e483323274aa1968278b79d1ec0f0"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "8c905cefe2513e76b01d6d603a0647d7"
server_key_iv: "95eb17f9bdaa152a56d5b165155232be"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.276813 14174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:26.285346 14174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:26.286880 14174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:26.286999 14174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:26.287175 14174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:26.287429 14174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:26.287460 14174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.287487 14174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:26.287503 14174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.292768 14174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44309
I20260928 23:19:26.292826 14302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44309 every 8 connection(s)
I20260928 23:19:26.293081 14174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:19:26 dist-test-slave-djkr krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:26.300483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 14174
I20260928 23:19:26.300583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:19:26.300788 392 external_mini_cluster.cc:1428] Setting key a6ba76c5c87b145c9a37474a102c6dfd
I20260928 23:19:26.301314 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.293657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52859 (local address 127.0.98.62:44711)
0928 23:19:26.293701 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.293703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.293908 (+ 205us) server_negotiation.cc:407] Connection header received
0928 23:19:26.294376 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.294377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.294390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.294435 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:26.295320 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.295848 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.296785 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.296981 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.297035 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.298909 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:26.298910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:26.298911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:26.298918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:26.300050 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:26.300397 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:26.300398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:26.300398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:26.300449 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:26.300700 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:26.300701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:26.300701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:26.300805 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:26.300866 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.301147 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.301234 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:19:26.301420 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.293639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:52859)
0928 23:19:26.293799 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:26.293831 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:26.294286 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:26.294514 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:26.294518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:26.295045 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:26.295229 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.295233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.295934 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.295934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.296708 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.296711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.297046 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.297047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.297161 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.297553 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:26.297561 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:26.298805 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:26.300116 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:26.300121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:26.300123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:26.300332 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:26.300507 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:26.300507 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:26.300508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:26.300638 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:26.300866 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:26.300872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:26.301010 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 23:19:26.301120 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 23:19:26.301237 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260928 23:19:26.301837 14303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44711
I20260928 23:19:26.301941 14303 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:26.302103 14303 heartbeater.cc:511] Master 127.0.98.62:44711 requested a full tablet report, sending...
I20260928 23:19:26.302575 14112 ts_manager.cc:194] Registered new tserver with Master: 7c9e483323274aa1968278b79d1ec0f0 (127.0.98.1:44309)
I20260928 23:19:26.303433 14112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52859
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:19:26.321977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44711
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:26.398665 14308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:26.398861 14308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:26.398882 14308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:26.399966 14308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:26.399996 14308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:26.400019 14308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:19:26.401336 14308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:26.401505 14308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:26.401678 14308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:26.406240 14316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:26.406212 14314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:26.406400 14308 server_base.cc:1034] running on GCE node
W20260928 23:19:26.406227 14313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:26.406636 14308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:26.406847 14308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:26.407984 14308 hybrid_clock.cc:648] HybridClock initialized: now 1790637566407969 us; error 29 us; skew 500 ppm
Sep 28 23:19:26 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:26.410239 14308 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:19:26.410661 14308 webserver.cc:460] Webserver started at http://127.0.98.2:37649/ using document root <none> and password file <none>
I20260928 23:19:26.410863 14308 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:26.410912 14308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:26.411026 14308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:26.412045 14308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "e4f8bdf5850148a19c0245d6d073f5e2"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "ffeb18cd2bcb79d783ecc05daa5389b5"
server_key_iv: "8b968ce5e91479684b5708735f3350f0"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.412365 14308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e4f8bdf5850148a19c0245d6d073f5e2"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "ffeb18cd2bcb79d783ecc05daa5389b5"
server_key_iv: "8b968ce5e91479684b5708735f3350f0"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.413563 14308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:19:26.414230 14323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.414389 14308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:19:26.414467 14308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "e4f8bdf5850148a19c0245d6d073f5e2"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "ffeb18cd2bcb79d783ecc05daa5389b5"
server_key_iv: "8b968ce5e91479684b5708735f3350f0"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.414526 14308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:26.426723 14308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:26.428371 14308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:26.428494 14308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:26.428686 14308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:26.428952 14308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:26.428984 14308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.429010 14308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:26.429033 14308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.434365 14308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44279
I20260928 23:19:26.434401 14436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44279 every 8 connection(s)
I20260928 23:19:26.434680 14308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:19:26.435798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 14308
I20260928 23:19:26.435896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:19:26.436100 392 external_mini_cluster.cc:1428] Setting key d5c132e701e153fda9c6ea778079a39f
Sep 28 23:19:26 dist-test-slave-djkr krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:19:26.443825 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.435305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40593 (local address 127.0.98.62:44711)
0928 23:19:26.435363 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.435366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.435567 (+ 201us) server_negotiation.cc:407] Connection header received
0928 23:19:26.436223 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.436224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.436236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.436300 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:26.437474 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.438015 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.438957 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.439165 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.439229 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.441372 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:26.441374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:26.441374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:26.441381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:26.442597 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:26.442942 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:26.442943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:26.442943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:26.442994 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:26.443192 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:26.443193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:26.443194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:26.443298 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:26.443351 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.443651 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.443736 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 23:19:26.443964 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.435286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:40593)
0928 23:19:26.435442 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:26.435476 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:26.436122 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:26.436365 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:26.436369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:26.437089 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:26.437363 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.437370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.438106 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.438107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.438880 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.438882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.439245 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.439246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.439369 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.439848 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:26.439856 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:26.441286 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:26.442667 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:26.442673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:26.442676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:26.442876 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:26.443045 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:26.443045 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:26.443046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:26.443140 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:26.443350 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:26.443356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:26.443509 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 23:19:26.443621 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 23:19:26.443764 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260928 23:19:26.444393 14437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44711
I20260928 23:19:26.444483 14437 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:26.444648 14437 heartbeater.cc:511] Master 127.0.98.62:44711 requested a full tablet report, sending...
I20260928 23:19:26.444985 14112 ts_manager.cc:194] Registered new tserver with Master: e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279)
I20260928 23:19:26.445384 14112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:19:26.458369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44711
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:26.536628 14442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:26.536834 14442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:26.536855 14442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:26.538005 14442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:26.538048 14442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:26.538072 14442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:19:26.539462 14442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:26.539690 14442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:26.539889 14442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:26.544483 14447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:26.544528 14448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:26.544487 14450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:26.544703 14442 server_base.cc:1034] running on GCE node
I20260928 23:19:26.544894 14442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:26.545096 14442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:26.546240 14442 hybrid_clock.cc:648] HybridClock initialized: now 1790637566546221 us; error 34 us; skew 500 ppm
Sep 28 23:19:26 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:26.548599 14442 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:19:26.549021 14442 webserver.cc:460] Webserver started at http://127.0.98.3:44613/ using document root <none> and password file <none>
I20260928 23:19:26.549206 14442 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:26.549252 14442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:26.549371 14442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:26.550314 14442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a08d21578524f4686c468e80c414e11"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "1ba8f16900a9bcc1ea42509c8179ab83"
server_key_iv: "02d0b2ba2bdb7e0c8a514c08491afb05"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.550625 14442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a08d21578524f4686c468e80c414e11"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "1ba8f16900a9bcc1ea42509c8179ab83"
server_key_iv: "02d0b2ba2bdb7e0c8a514c08491afb05"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.551791 14442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:19:26.552428 14457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.552565 14442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:19:26.552640 14442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "4a08d21578524f4686c468e80c414e11"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "1ba8f16900a9bcc1ea42509c8179ab83"
server_key_iv: "02d0b2ba2bdb7e0c8a514c08491afb05"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.552703 14442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:26.567314 14442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:26.569343 14442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:26.569471 14442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:26.569669 14442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:26.569948 14442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:26.569979 14442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.570008 14442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:26.570032 14442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.575228 14442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46523
I20260928 23:19:26.575266 14570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46523 every 8 connection(s)
I20260928 23:19:26.575541 14442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:19:26 dist-test-slave-djkr krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:26.582563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 14442
I20260928 23:19:26.582647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:19:26.582839 392 external_mini_cluster.cc:1428] Setting key 3182db432a8396ebc0687ab6ab5381a9
I20260928 23:19:26.584244 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.576155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48425 (local address 127.0.98.62:44711)
0928 23:19:26.576198 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.576200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.576399 (+ 199us) server_negotiation.cc:407] Connection header received
0928 23:19:26.576902 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.576903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.576915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.576965 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:26.577838 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.578332 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.579266 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.579493 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.579563 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.581475 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:26.581477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:26.581477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:26.581483 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:26.582620 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:26.583118 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:26.583120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:26.583120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:26.583187 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:26.583598 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:26.583599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:26.583600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:26.583742 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:26.583799 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.584118 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.584183 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:19:26.584383 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.576131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:48425)
0928 23:19:26.576292 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:26.576327 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:26.576807 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:26.577041 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:26.577045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:26.577573 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:26.577748 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.577753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.578409 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.578410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.579187 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.579189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.579605 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.579606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.579717 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.580137 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:26.580147 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:26.581396 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:26.582717 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:26.582724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:26.582726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:26.582997 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:26.583259 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:26.583260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:26.583260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:26.583478 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:26.583825 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:26.583832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:26.583991 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 23:19:26.584078 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:19:26.584196 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260928 23:19:26.584765 14571 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44711
I20260928 23:19:26.584838 14571 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:26.584970 14571 heartbeater.cc:511] Master 127.0.98.62:44711 requested a full tablet report, sending...
I20260928 23:19:26.585289 14112 ts_manager.cc:194] Registered new tserver with Master: 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523)
I20260928 23:19:26.585620 14112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48425
I20260928 23:19:26.594655 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:19:26.601212 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.595905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.0.98.62:44711)
0928 23:19:26.595962 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.595964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.596040 (+ 76us) server_negotiation.cc:407] Connection header received
0928 23:19:26.596079 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.596080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.596087 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.596132 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:26.596610 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.597080 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.597746 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.597941 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.597988 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.598982 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:26.598983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:26.598984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:26.598990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:26.600299 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:26.600571 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:26.600572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:26.600572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:26.600626 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:26.600811 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:26.600812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:26.600812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:26.600946 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:26.601012 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.601099 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.601152 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:19:26.602594 14112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:19:26.604182 14112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:19:26.610154 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.606238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46523 (local address 127.0.0.1:40890)
0928 23:19:26.606474 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:26.606484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:26.606537 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:26.606943 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:26.606944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:26.606946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:26.606996 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.606997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.608082 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.608083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.609134 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.609137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.609887 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.609888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.609992 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.610006 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:19:26.610063 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:19:26.610100 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":68,"thread_start_us":46,"threads_started":1}
I20260928 23:19:26.610208 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.606625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.0.98.3:46523)
0928 23:19:26.606770 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.606774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.606785 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:19:26.606824 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.606826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.606841 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.607063 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:26.607126 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.607973 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.609202 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.609809 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.609871 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.609928 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.610092 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.610132 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260928 23:19:26.610787 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.606238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44309 (local address 127.0.0.1:47590)
0928 23:19:26.606401 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:26.606436 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:26.606498 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:26.606860 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:26.606864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:26.606866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:26.606925 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.606927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.608192 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.608192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.609362 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.609364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.610178 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.610179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.610630 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.610634 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:19:26.610684 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:19:26.610724 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20260928 23:19:26.610826 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.606436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.0.98.1:44309)
0928 23:19:26.606700 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.606704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.606713 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:19:26.606745 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.606747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.606758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.606826 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:26.607005 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.608101 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.609454 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.610105 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.610167 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.610219 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.610715 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.610755 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":92,"threads_started":1}
I20260928 23:19:26.611186 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.606259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44279 (local address 127.0.0.1:49618)
0928 23:19:26.606770 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:26.606779 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:26.606829 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:26.607099 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:26.607099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:26.607101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:26.607136 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.607137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.608326 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.608327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.609371 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.609373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.610970 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.610971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.611031 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.611035 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:19:26.611087 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:19:26.611123 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":76,"threads_started":1}
I20260928 23:19:26.611075 14505 tablet_service.cc:1467] Processing CreateTablet for tablet fc038d0a820c41909a816f47300b70d4 (DEFAULT_TABLE table=test-table [id=ed2378ac843047d2b16d325fd8ad539e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:26.611426 14505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc038d0a820c41909a816f47300b70d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:26.611438 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.606497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.0.98.2:44279)
0928 23:19:26.606643 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.606648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.606902 (+ 254us) server_negotiation.cc:407] Connection header received
0928 23:19:26.606941 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.606943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.606956 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.607022 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:26.607198 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.608247 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.610252 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.610862 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.610960 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.611013 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.611330 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.611376 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260928 23:19:26.611644 14237 tablet_service.cc:1467] Processing CreateTablet for tablet fc038d0a820c41909a816f47300b70d4 (DEFAULT_TABLE table=test-table [id=ed2378ac843047d2b16d325fd8ad539e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:26.611945 14237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc038d0a820c41909a816f47300b70d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:26.612074 14371 tablet_service.cc:1467] Processing CreateTablet for tablet fc038d0a820c41909a816f47300b70d4 (DEFAULT_TABLE table=test-table [id=ed2378ac843047d2b16d325fd8ad539e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:26.612309 14371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc038d0a820c41909a816f47300b70d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:26.614678 14585 tablet_bootstrap.cc:492] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: Bootstrap starting.
I20260928 23:19:26.614960 14586 tablet_bootstrap.cc:492] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: Bootstrap starting.
I20260928 23:19:26.615068 14587 tablet_bootstrap.cc:492] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Bootstrap starting.
I20260928 23:19:26.615415 14585 tablet_bootstrap.cc:654] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:26.615641 14587 tablet_bootstrap.cc:654] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:26.615641 14586 tablet_bootstrap.cc:654] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:26.615901 14585 log.cc:824] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:26.616159 14586 log.cc:824] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:26.616159 14587 log.cc:824] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:26.616539 14585 tablet_bootstrap.cc:492] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: No bootstrap required, opened a new log
I20260928 23:19:26.616604 14585 ts_tablet_manager.cc:1397] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:19:26.616829 14587 tablet_bootstrap.cc:492] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: No bootstrap required, opened a new log
I20260928 23:19:26.616865 14586 tablet_bootstrap.cc:492] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: No bootstrap required, opened a new log
I20260928 23:19:26.616914 14587 ts_tablet_manager.cc:1397] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:19:26.616917 14586 ts_tablet_manager.cc:1397] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:19:26.618351 14586 raft_consensus.cc:348] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
I20260928 23:19:26.618470 14586 raft_consensus.cc:374] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:26.618489 14586 raft_consensus.cc:729] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f8bdf5850148a19c0245d6d073f5e2, State: Initialized, Role: FOLLOWER
I20260928 23:19:26.618557 14586 consensus_queue.cc:260] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
I20260928 23:19:26.618505 14585 raft_consensus.cc:348] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
I20260928 23:19:26.618661 14585 raft_consensus.cc:374] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:26.618697 14585 raft_consensus.cc:729] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a08d21578524f4686c468e80c414e11, State: Initialized, Role: FOLLOWER
I20260928 23:19:26.618681 14587 raft_consensus.cc:348] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
I20260928 23:19:26.618764 14586 ts_tablet_manager.cc:1428] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:19:26.618821 14587 raft_consensus.cc:374] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:26.618798 14437 heartbeater.cc:503] Master 127.0.98.62:44711 was elected leader, sending a full tablet report...
I20260928 23:19:26.618808 14585 consensus_queue.cc:260] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
I20260928 23:19:26.618850 14587 raft_consensus.cc:729] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9e483323274aa1968278b79d1ec0f0, State: Initialized, Role: FOLLOWER
I20260928 23:19:26.618973 14587 consensus_queue.cc:260] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
I20260928 23:19:26.619052 14585 ts_tablet_manager.cc:1428] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 23:19:26.619094 14571 heartbeater.cc:503] Master 127.0.98.62:44711 was elected leader, sending a full tablet report...
I20260928 23:19:26.619194 14587 ts_tablet_manager.cc:1428] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:19:26.619294 14303 heartbeater.cc:503] Master 127.0.98.62:44711 was elected leader, sending a full tablet report...
I20260928 23:19:26.677307 14593 raft_consensus.cc:482] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:19:26.677443 14593 raft_consensus.cc:504] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
I20260928 23:19:26.677789 14593 leader_election.cc:288] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523), e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279)
I20260928 23:19:26.681646 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.677957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46523 (local address 127.0.98.1:58203)
0928 23:19:26.678043 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:26.678052 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:26.678091 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:26.678326 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:26.678327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:26.678329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:26.678423 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.678424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.679309 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.679310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.680685 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.680688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.681411 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.681412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.681493 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.681498 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:19:26.681542 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:19:26.681575 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":78}
I20260928 23:19:26.681668 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.678153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58203 (local address 127.0.98.3:46523)
0928 23:19:26.678214 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.678215 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.678223 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:19:26.678249 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.678250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.678261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.678312 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:26.678485 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.679245 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.680764 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.681342 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.681417 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.681426 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.681566 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.681605 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:19:26.681721 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.678055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52141 (local address 127.0.98.2:44279)
0928 23:19:26.678108 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.678110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.678204 (+ 94us) server_negotiation.cc:407] Connection header received
0928 23:19:26.678255 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.678256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.678266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.678317 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:26.678491 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.679245 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.680779 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.681342 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.681410 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.681422 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.681619 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.681662 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:19:26.681833 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.677966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44279 (local address 127.0.98.1:52141)
0928 23:19:26.678138 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:26.678148 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:26.678210 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:26.678329 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:26.678330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:26.678332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:26.678423 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.678424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.679309 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.679310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.680685 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:26.680688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.681471 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.681472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:26.681541 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.681545 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:19:26.681599 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 23:19:26.681634 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":78,"thread_start_us":93,"threads_started":1}
I20260928 23:19:26.682027 14525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc038d0a820c41909a816f47300b70d4" candidate_uuid: "7c9e483323274aa1968278b79d1ec0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a08d21578524f4686c468e80c414e11" is_pre_election: true
I20260928 23:19:26.682185 14525 raft_consensus.cc:2454] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c9e483323274aa1968278b79d1ec0f0 in term 0.
I20260928 23:19:26.682408 14192 leader_election.cc:302] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a08d21578524f4686c468e80c414e11, 7c9e483323274aa1968278b79d1ec0f0; no voters:
I20260928 23:19:26.682405 14391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc038d0a820c41909a816f47300b70d4" candidate_uuid: "7c9e483323274aa1968278b79d1ec0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" is_pre_election: true
I20260928 23:19:26.682539 14391 raft_consensus.cc:2454] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c9e483323274aa1968278b79d1ec0f0 in term 0.
I20260928 23:19:26.682585 14593 raft_consensus.cc:2781] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:19:26.682657 14593 raft_consensus.cc:482] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:19:26.682685 14593 raft_consensus.cc:3037] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:26.683385 14593 raft_consensus.cc:504] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
I20260928 23:19:26.683511 14593 leader_election.cc:288] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523), e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279)
I20260928 23:19:26.683667 14391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc038d0a820c41909a816f47300b70d4" candidate_uuid: "7c9e483323274aa1968278b79d1ec0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f8bdf5850148a19c0245d6d073f5e2"
I20260928 23:19:26.683708 14525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc038d0a820c41909a816f47300b70d4" candidate_uuid: "7c9e483323274aa1968278b79d1ec0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a08d21578524f4686c468e80c414e11"
I20260928 23:19:26.683749 14391 raft_consensus.cc:3037] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:26.683771 14525 raft_consensus.cc:3037] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:26.684413 14391 raft_consensus.cc:2454] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c9e483323274aa1968278b79d1ec0f0 in term 1.
I20260928 23:19:26.684403 14525 raft_consensus.cc:2454] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c9e483323274aa1968278b79d1ec0f0 in term 1.
I20260928 23:19:26.684561 14192 leader_election.cc:302] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a08d21578524f4686c468e80c414e11, 7c9e483323274aa1968278b79d1ec0f0; no voters:
I20260928 23:19:26.684630 14593 raft_consensus.cc:2781] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:19:26.684798 14593 raft_consensus.cc:686] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 LEADER]: Becoming Leader. State: Replica: 7c9e483323274aa1968278b79d1ec0f0, State: Running, Role: LEADER
I20260928 23:19:26.684892 14593 consensus_queue.cc:237] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
W20260928 23:19:26.685150 14438 tablet.cc:2369] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:19:26.685708 14112 catalog_manager.cc:5495] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c9e483323274aa1968278b79d1ec0f0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7c9e483323274aa1968278b79d1ec0f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } health_report { overall_health: UNKNOWN } } }
I20260928 23:19:26.704190 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.702177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47602 (local address 127.0.98.1:44309)
0928 23:19:26.702224 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:26.702225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:26.702236 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:19:26.702269 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:26.702270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:26.702287 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:26.702341 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:19:26.702444 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.702888 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.703624 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:26.703807 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:26.703851 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:26.703956 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:19:26.704051 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:19:26.704105 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:26.704120 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 23:19:26.704137 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:19:26.705811 14525 raft_consensus.cc:1261] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11 [term 1 FOLLOWER]: Refusing update from remote peer 7c9e483323274aa1968278b79d1ec0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:19:26.705865 14391 raft_consensus.cc:1261] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2 [term 1 FOLLOWER]: Refusing update from remote peer 7c9e483323274aa1968278b79d1ec0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:19:26.706127 14595 consensus_queue.cc:1035] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:19:26.706249 14593 consensus_queue.cc:1035] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:19:26.709060 14599 mvcc.cc:205] Tried to move back new op lower bound from 7334451473224613888 to 7334451473141948416. Current Snapshot: MvccSnapshot[applied={T|T < 7334451473224613888}]
I20260928 23:19:26.709182 14600 mvcc.cc:205] Tried to move back new op lower bound from 7334451473224613888 to 7334451473141948416. Current Snapshot: MvccSnapshot[applied={T|T < 7334451473224613888 or (T in {7334451473224613888})}]
I20260928 23:19:26.710382 14111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:19:26.710466 14111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:19:26.711150 14111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Sending DeleteTablet for 3 replicas of tablet fc038d0a820c41909a816f47300b70d4
I20260928 23:19:26.711421 14505 tablet_service.cc:1514] Processing DeleteTablet for tablet fc038d0a820c41909a816f47300b70d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40890
I20260928 23:19:26.711463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 14174
I20260928 23:19:26.711449 14371 tablet_service.cc:1514] Processing DeleteTablet for tablet fc038d0a820c41909a816f47300b70d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49618
I20260928 23:19:26.711659 14609 tablet_replica.cc:330] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: stopping tablet replica
I20260928 23:19:26.711772 14609 raft_consensus.cc:2229] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:19:26.711941 14609 raft_consensus.cc:2258] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:19:26.712311 14609 ts_tablet_manager.cc:1905] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:19:26.713821 14609 ts_tablet_manager.cc:1918] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:19:26.713879 14609 log.cc:1196] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/fc038d0a820c41909a816f47300b70d4
I20260928 23:19:26.714113 14609 ts_tablet_manager.cc:1939] T fc038d0a820c41909a816f47300b70d4 P e4f8bdf5850148a19c0245d6d073f5e2: Deleting consensus metadata
I20260928 23:19:26.714454 14099 catalog_manager.cc:4841] TS e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279): tablet fc038d0a820c41909a816f47300b70d4 (table test-table [id=ed2378ac843047d2b16d325fd8ad539e]) successfully deleted
W20260928 23:19:26.716864 14097 connection.cc:535] client connection to 127.0.98.1:44309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 23:19:26.716933 14097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:19:26.717077 14608 tablet_replica.cc:330] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: stopping tablet replica
I20260928 23:19:26.717170 14608 raft_consensus.cc:2229] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:19:26.717273 14608 raft_consensus.cc:2258] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:19:26.717299 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:26.717195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44309 (local address 127.0.0.1:47610)
0928 23:19:26.717235 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:26.717269 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 23:19:26.717438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 14308
W20260928 23:19:26.717484 14097 catalog_manager.cc:4584] TS 7c9e483323274aa1968278b79d1ec0f0 (127.0.98.1:44309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc038d0a820c41909a816f47300b70d4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44309: connect: Connection refused (error 111)
I20260928 23:19:26.718205 14608 ts_tablet_manager.cc:1905] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:19:26.719496 14608 ts_tablet_manager.cc:1918] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:19:26.719568 14608 log.cc:1196] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/fc038d0a820c41909a816f47300b70d4
I20260928 23:19:26.719775 14608 ts_tablet_manager.cc:1939] T fc038d0a820c41909a816f47300b70d4 P 4a08d21578524f4686c468e80c414e11: Deleting consensus metadata
I20260928 23:19:26.720108 14098 catalog_manager.cc:4841] TS 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523): tablet fc038d0a820c41909a816f47300b70d4 (table test-table [id=ed2378ac843047d2b16d325fd8ad539e]) successfully deleted
I20260928 23:19:26.721834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 14442
I20260928 23:19:26.725925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 14080
I20260928 23:19:26.729971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44711
--webserver_interface=127.0.98.62
--webserver_port=35989
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:26.807693 14611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:26.807896 14611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:26.807919 14611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:26.809173 14611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:19:26.809221 14611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:26.809234 14611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:26.809255 14611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:19:26.809270 14611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:19:26.810777 14611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:26.811017 14611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:26.811239 14611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:26.819679 14617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:26.819680 14618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:26.819680 14620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:26.819710 14611 server_base.cc:1034] running on GCE node
I20260928 23:19:26.820138 14611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:26.820423 14611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:26.821564 14611 hybrid_clock.cc:648] HybridClock initialized: now 1790637566821547 us; error 35 us; skew 500 ppm
Sep 28 23:19:26 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:26.823947 14611 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:19:26.824383 14611 webserver.cc:460] Webserver started at http://127.0.98.62:35989/ using document root <none> and password file <none>
I20260928 23:19:26.824594 14611 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:26.824648 14611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:26.825924 14611 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:26.826471 14627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.826615 14611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.826694 14611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "898107bef00d473c842ff384f491c3b2"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "01225fa8211d30a90cbd0a7470ba0c08"
server_key_iv: "298a70ab06c5a7594bd861bbe5d1edba"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.827066 14611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:26.849234 14611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:26.851061 14611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:26.851189 14611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:26.854875 14611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44711
I20260928 23:19:26.854923 14679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44711 every 8 connection(s)
I20260928 23:19:26.855257 14611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:19:26.856783 14680 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:19:26.857411 14680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Bootstrap starting.
I20260928 23:19:26.864439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 14611
I20260928 23:19:26.864673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44309
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44711
--webserver_port=38129
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:19:26.865248 14680 log.cc:824] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:26.867403 14680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:19:26.867697 14680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Bootstrap complete.
I20260928 23:19:26.869797 14680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:26.870020 14680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898107bef00d473c842ff384f491c3b2, State: Initialized, Role: FOLLOWER
I20260928 23:19:26.870182 14680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:26.870258 14680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:19:26.870311 14680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:19:26.870355 14680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:19:26.871320 14680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:26.871414 14680 leader_election.cc:302] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 898107bef00d473c842ff384f491c3b2; no voters:
I20260928 23:19:26.871587 14680 leader_election.cc:288] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:19:26.871627 14684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:19:26.871733 14684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 2 LEADER]: Becoming Leader. State: Replica: 898107bef00d473c842ff384f491c3b2, State: Running, Role: LEADER
I20260928 23:19:26.871842 14680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:19:26.871872 14684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:26.872241 14686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 898107bef00d473c842ff384f491c3b2. Latest consensus state: current_term: 2 leader_uuid: "898107bef00d473c842ff384f491c3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } } }
I20260928 23:19:26.872326 14686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:26.872468 14685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "898107bef00d473c842ff384f491c3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } } }
I20260928 23:19:26.872525 14685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:26.872711 14691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:19:26.873564 14691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:19:26.873659 14691 catalog_manager.cc:1259] Loaded cluster ID: 2adeaec8dbb34980b3a4fbb59b0b927d
I20260928 23:19:26.873687 14691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:19:26.874060 14691 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 23:19:26.879269 14691 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 23:19:26.881683 14691 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:19:26.881873 14691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Loaded TSK: 0
I20260928 23:19:26.882078 14691 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:19:26.945961 14683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:26.946149 14683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:26.946170 14683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:26.947299 14683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:26.947336 14683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:26.947362 14683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:19:26.948832 14683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38129
--tserver_master_addrs=127.0.98.62:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:26.949038 14683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:26.949254 14683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:26.953866 14710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:26.953989 14683 server_base.cc:1034] running on GCE node
W20260928 23:19:26.953841 14708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:26.953874 14707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:26.954312 14683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:26.954541 14683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:26.955680 14683 hybrid_clock.cc:648] HybridClock initialized: now 1790637566955670 us; error 26 us; skew 500 ppm
Sep 28 23:19:26 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:26.958057 14683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:19:26.958469 14683 webserver.cc:460] Webserver started at http://127.0.98.1:38129/ using document root <none> and password file <none>
I20260928 23:19:26.958670 14683 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:26.958729 14683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:26.960054 14683 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:19:26.960651 14717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:26.960806 14683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:19:26.960907 14683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "7c9e483323274aa1968278b79d1ec0f0"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "8c905cefe2513e76b01d6d603a0647d7"
server_key_iv: "95eb17f9bdaa152a56d5b165155232be"
server_key_version: "encryptionkey@0"
I20260928 23:19:26.961218 14683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:26.989722 14683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:26.991678 14683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:26.991793 14683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:26.992012 14683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:26.992417 14724 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:19:26.993821 14683 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:19:26.993865 14683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 23:19:26.993902 14683 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:19:26.994511 14683 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:19:26.994540 14683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 23:19:26.994607 14724 tablet_bootstrap.cc:492] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Bootstrap starting.
I20260928 23:19:27.000025 14683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44309
I20260928 23:19:27.000098 14831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44309 every 8 connection(s)
I20260928 23:19:27.000438 14683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:19:27.003978 14724 log.cc:824] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:27.006040 14724 tablet_bootstrap.cc:492] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:19:27.006328 14724 tablet_bootstrap.cc:492] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Bootstrap complete.
Sep 28 23:19:27 dist-test-slave-djkr krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:27.007349 14724 ts_tablet_manager.cc:1397] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260928 23:19:27.008450 14724 raft_consensus.cc:348] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
I20260928 23:19:27.008735 14724 raft_consensus.cc:729] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9e483323274aa1968278b79d1ec0f0, State: Initialized, Role: FOLLOWER
I20260928 23:19:27.008886 14724 consensus_queue.cc:260] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } }
I20260928 23:19:27.009156 14724 ts_tablet_manager.cc:1428] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260928 23:19:27.009805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 14683
I20260928 23:19:27.010041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44279
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44711
--webserver_port=37649
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:19:27.010726 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.001083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:39885)
0928 23:19:27.001249 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.001289 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:27.001770 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:27.002159 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:27.002163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:27.002433 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:27.002613 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.002617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.003745 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.003747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.004568 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.004571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.005144 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.005145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.005266 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.005775 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:27.005783 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:27.007448 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:27.009253 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:27.009258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:27.009260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:27.009472 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:27.009693 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:27.009694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:27.009694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:27.009793 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:27.010181 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:27.010187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:27.010329 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 23:19:27.010430 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 23:19:27.010585 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260928 23:19:27.010846 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.001198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39885 (local address 127.0.98.62:44711)
0928 23:19:27.001383 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.001391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.001435 (+ 44us) server_negotiation.cc:407] Connection header received
0928 23:19:27.001970 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.001974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.002016 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.002126 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:27.002719 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.003633 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.004645 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.005050 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.005145 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.007569 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:27.007574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:27.007574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:27.007586 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:27.009183 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:27.009541 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:27.009544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:27.009545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:27.009637 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:27.009854 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:27.009855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:27.009855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:27.010086 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:27.010154 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.010527 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.010669 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260928 23:19:27.011555 14832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44711
I20260928 23:19:27.011647 14832 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:27.011802 14832 heartbeater.cc:511] Master 127.0.98.62:44711 requested a full tablet report, sending...
I20260928 23:19:27.012449 14644 ts_manager.cc:194] Registered new tserver with Master: 7c9e483323274aa1968278b79d1ec0f0 (127.0.98.1:44309)
I20260928 23:19:27.013540 14644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39885
I20260928 23:19:27.014379 14832 heartbeater.cc:503] Master 127.0.98.62:44711 was elected leader, sending a full tablet report...
Sep 28 23:19:27 dist-test-slave-djkr krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:19:27.020774 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.013208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.0.98.1:44309)
0928 23:19:27.013335 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.013339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.013348 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:19:27.013378 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.013380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.013395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.013467 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:27.013945 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.014866 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.015930 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.016296 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.016354 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.018281 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:27.018289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:27.018289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:27.018296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:27.019675 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:27.020009 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:27.020012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:27.020013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:27.020063 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:27.020223 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:27.020224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:27.020224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:27.020355 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:27.020422 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.020604 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.020687 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260928 23:19:27.020774 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.013016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44309 (local address 127.0.0.1:47622)
0928 23:19:27.013184 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.013224 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:27.013286 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:27.013476 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:27.013480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:27.013797 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:27.013868 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.013870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.014938 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.014940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.015858 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.015861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.016367 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.016371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.016452 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.016860 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:27.016865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:27.018204 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:27.019743 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:27.019749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:27.019751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:27.019933 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:27.020118 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:27.020119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:27.020119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:27.020175 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:27.020426 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:27.020433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:27.020483 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 23:19:27.020561 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:19:27.020675 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260928 23:19:27.021194 14766 tablet_service.cc:1514] Processing DeleteTablet for tablet fc038d0a820c41909a816f47300b70d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47622
I20260928 23:19:27.021736 14841 tablet_replica.cc:330] stopping tablet replica
I20260928 23:19:27.021831 14841 raft_consensus.cc:2229] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:19:27.021894 14841 raft_consensus.cc:2258] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:19:27.022149 14841 ts_tablet_manager.cc:1905] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:19:27.024037 14841 ts_tablet_manager.cc:1918] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:19:27.024101 14841 log.cc:1196] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/fc038d0a820c41909a816f47300b70d4
I20260928 23:19:27.024274 14841 ts_tablet_manager.cc:1939] T fc038d0a820c41909a816f47300b70d4 P 7c9e483323274aa1968278b79d1ec0f0: Deleting consensus metadata
I20260928 23:19:27.024700 14629 catalog_manager.cc:4841] TS 7c9e483323274aa1968278b79d1ec0f0 (127.0.98.1:44309): tablet fc038d0a820c41909a816f47300b70d4 (table test-table [id=ed2378ac843047d2b16d325fd8ad539e]) successfully deleted
W20260928 23:19:27.089686 14838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:27.089874 14838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:27.089895 14838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:27.091009 14838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:27.091046 14838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:27.091070 14838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:19:27.092484 14838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37649
--tserver_master_addrs=127.0.98.62:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:27.092686 14838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:27.092907 14838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:27.097589 14849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:27.097656 14846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:27.097591 14847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:27.097743 14838 server_base.cc:1034] running on GCE node
I20260928 23:19:27.097934 14838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:27.098155 14838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:27.099294 14838 hybrid_clock.cc:648] HybridClock initialized: now 1790637567099277 us; error 31 us; skew 500 ppm
Sep 28 23:19:27 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:27.101608 14838 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:19:27.102023 14838 webserver.cc:460] Webserver started at http://127.0.98.2:37649/ using document root <none> and password file <none>
I20260928 23:19:27.102211 14838 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:27.102270 14838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:27.103582 14838 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:27.104133 14856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.104264 14838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.104362 14838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "e4f8bdf5850148a19c0245d6d073f5e2"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "ffeb18cd2bcb79d783ecc05daa5389b5"
server_key_iv: "8b968ce5e91479684b5708735f3350f0"
server_key_version: "encryptionkey@0"
I20260928 23:19:27.104678 14838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:27.120716 14838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:27.122332 14838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:27.122459 14838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:27.122651 14838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:27.122911 14838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:27.122942 14838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.122970 14838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:27.123001 14838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.128257 14838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44279
I20260928 23:19:27.128305 14969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44279 every 8 connection(s)
I20260928 23:19:27.128604 14838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:19:27 dist-test-slave-djkr krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:27.134848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 14838
I20260928 23:19:27.135092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46523
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44711
--webserver_port=44613
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:19:27.137326 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.129255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35007 (local address 127.0.98.62:44711)
0928 23:19:27.129296 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.129298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.129536 (+ 238us) server_negotiation.cc:407] Connection header received
0928 23:19:27.130081 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.130082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.130098 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.130158 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:27.131071 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.131588 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.132525 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.132717 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.132769 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.134643 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:27.134644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:27.134645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:27.134652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:27.136057 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:27.136464 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:27.136465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:27.136465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:27.136521 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:27.136754 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:27.136754 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:27.136754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:27.136854 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:27.136897 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.137189 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.137255 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:19:27.137547 14972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.129262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:35007)
0928 23:19:27.129427 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.129460 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:27.129981 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:27.130236 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:27.130240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:27.130783 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:27.130974 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.130979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.131659 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.131660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.132443 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.132446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.132780 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.132781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.132897 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.133292 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:27.133300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:27.134554 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:27.136129 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:27.136135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:27.136137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:27.136375 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:27.136573 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:27.136574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:27.136575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:27.136701 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:27.136901 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:27.136905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:27.137067 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 23:19:27.137170 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 23:19:27.137336 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260928 23:19:27.137988 14970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44711
I20260928 23:19:27.138074 14970 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:27.138242 14970 heartbeater.cc:511] Master 127.0.98.62:44711 requested a full tablet report, sending...
I20260928 23:19:27.138597 14644 ts_manager.cc:194] Registered new tserver with Master: e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279)
I20260928 23:19:27.139060 14644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35007
W20260928 23:19:27.214655 14973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:27.214852 14973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:27.214874 14973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:27.216034 14973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:27.216073 14973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:27.216097 14973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:19:27.217536 14973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44613
--tserver_master_addrs=127.0.98.62:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:27.217721 14973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:27.217896 14973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:27.222545 14978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:27.222551 14981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:27.222697 14979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:27.222575 14973 server_base.cc:1034] running on GCE node
I20260928 23:19:27.222924 14973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:27.223140 14973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:27.224300 14973 hybrid_clock.cc:648] HybridClock initialized: now 1790637567224287 us; error 27 us; skew 500 ppm
Sep 28 23:19:27 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:27.226696 14973 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:19:27.227111 14973 webserver.cc:460] Webserver started at http://127.0.98.3:44613/ using document root <none> and password file <none>
I20260928 23:19:27.227319 14973 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:27.227380 14973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:27.228658 14973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:27.229231 14988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.229388 14973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:19:27.229467 14973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "4a08d21578524f4686c468e80c414e11"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "1ba8f16900a9bcc1ea42509c8179ab83"
server_key_iv: "02d0b2ba2bdb7e0c8a514c08491afb05"
server_key_version: "encryptionkey@0"
I20260928 23:19:27.229777 14973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:27.252365 14973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:27.254153 14973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:27.254292 14973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:27.254494 14973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:27.254776 14973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:27.254808 14973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.254837 14973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:27.254859 14973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.259971 14973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46523
I20260928 23:19:27.260013 15101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46523 every 8 connection(s)
I20260928 23:19:27.260326 14973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:19:27.260457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 14973
Sep 28 23:19:27 dist-test-slave-djkr krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:27.267376 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.260844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.0.98.62:44711)
0928 23:19:27.260908 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.260910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.260969 (+ 59us) server_negotiation.cc:407] Connection header received
0928 23:19:27.261048 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.261049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.261061 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.261125 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:27.261640 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.262162 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.263290 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.263594 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.263662 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.265035 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:27.265036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:27.265037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:27.265043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:27.266319 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:27.266652 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:27.266653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:27.266654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:27.266712 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:27.266867 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:27.266867 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:27.266868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:27.267014 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:27.267178 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.267216 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.267287 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:19:27.269886 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.260989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35243 (local address 127.0.98.62:44711)
0928 23:19:27.261082 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.261084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.261782 (+ 698us) server_negotiation.cc:407] Connection header received
0928 23:19:27.262227 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.262228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.262241 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.262301 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:27.263394 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.264017 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.265349 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.265537 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.265588 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.267705 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:27.267706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:27.267707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:27.267713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:27.268940 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:27.269261 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:27.269261 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:27.269262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:27.269302 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:27.269463 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:27.269463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:27.269463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:27.269545 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:27.269587 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.269773 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.269825 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":46,"threads_started":1}
I20260928 23:19:27.270035 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.260990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:35243)
0928 23:19:27.261679 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.261713 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:27.262150 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:27.262373 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:27.262377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:27.263056 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:27.263337 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.263343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.264096 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.264098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.265276 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.265278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.265601 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.265601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.265720 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.266074 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:27.266082 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:27.267630 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:27.269001 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:27.269006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:27.269008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:27.269207 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:27.269343 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:27.269343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:27.269343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:27.269424 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:27.269585 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:27.269588 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:27.269690 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 23:19:27.269762 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:19:27.269867 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":79,"threads_started":1}
I20260928 23:19:27.270412 15102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44711
I20260928 23:19:27.270494 15102 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:27.270649 15102 heartbeater.cc:511] Master 127.0.98.62:44711 requested a full tablet report, sending...
I20260928 23:19:27.270949 14644 ts_manager.cc:194] Registered new tserver with Master: 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523)
I20260928 23:19:27.271265 14644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35243
I20260928 23:19:27.279263 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:19:27.281685 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.279541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.0.98.62:44711)
0928 23:19:27.279610 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.279611 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.279687 (+ 76us) server_negotiation.cc:407] Connection header received
0928 23:19:27.279732 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.279733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.279758 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.279810 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:19:27.279893 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.280332 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.281022 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.281223 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.281284 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.281434 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:19:27.281524 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:19:27.281590 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.281605 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.281625 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:19:27.281932 14644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:19:27.283483 14644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:19:27.285463 14766 tablet_service.cc:1467] Processing CreateTablet for tablet b91d5d043b914d4290d0c45548e8d071 (DEFAULT_TABLE table=test-table [id=22c211130d5042c1b5c081ac2224c9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:27.285652 14766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91d5d043b914d4290d0c45548e8d071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:27.286887 14724 tablet_bootstrap.cc:492] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Bootstrap starting.
I20260928 23:19:27.287199 14724 tablet_bootstrap.cc:654] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:27.288048 14724 tablet_bootstrap.cc:492] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: No bootstrap required, opened a new log
I20260928 23:19:27.288089 14724 ts_tablet_manager.cc:1397] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:19:27.288223 14724 raft_consensus.cc:348] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.288297 14724 raft_consensus.cc:374] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:27.288340 14724 raft_consensus.cc:729] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9e483323274aa1968278b79d1ec0f0, State: Initialized, Role: FOLLOWER
I20260928 23:19:27.288384 14724 consensus_queue.cc:260] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.288470 14724 ts_tablet_manager.cc:1428] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 23:19:27.289655 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.285225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46523 (local address 127.0.0.1:40898)
0928 23:19:27.285277 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.285287 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:27.285322 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:27.285735 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:27.285736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:27.285738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:27.285777 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.285779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.286740 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.286741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.287849 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.287852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.289416 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.289417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.289482 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.289500 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:19:27.289545 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:19:27.289588 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 23:19:27.289669 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.285429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.0.98.3:46523)
0928 23:19:27.285571 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.285575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.285585 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:19:27.285620 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.285622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.285637 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.285710 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:27.285855 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.286644 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.288659 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.289354 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.289410 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.289469 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.289558 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.289593 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260928 23:19:27.290119 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.285225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44279 (local address 127.0.0.1:49628)
0928 23:19:27.285402 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.285410 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:27.285442 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:27.287117 (+ 1675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:27.287118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:27.287120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:27.287160 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.287162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.288145 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.288146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.289194 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.289196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.289918 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.289919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.289987 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.289991 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:19:27.290046 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 23:19:27.290073 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":19,"thread_start_us":118,"threads_started":1}
I20260928 23:19:27.290182 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.285380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.0.98.2:44279)
0928 23:19:27.286946 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.286951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.286962 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:19:27.286998 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.287000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.287015 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.287115 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:27.287212 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.288050 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.289267 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.289843 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.289909 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.289961 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.290062 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.290102 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1525,"thread_start_us":83,"threads_started":1}
I20260928 23:19:27.290365 15036 tablet_service.cc:1467] Processing CreateTablet for tablet b91d5d043b914d4290d0c45548e8d071 (DEFAULT_TABLE table=test-table [id=22c211130d5042c1b5c081ac2224c9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:27.290709 15036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91d5d043b914d4290d0c45548e8d071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:27.290730 14904 tablet_service.cc:1467] Processing CreateTablet for tablet b91d5d043b914d4290d0c45548e8d071 (DEFAULT_TABLE table=test-table [id=22c211130d5042c1b5c081ac2224c9b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:27.290972 14904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91d5d043b914d4290d0c45548e8d071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:27.293219 15112 tablet_bootstrap.cc:492] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: Bootstrap starting.
I20260928 23:19:27.293448 15113 tablet_bootstrap.cc:492] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: Bootstrap starting.
I20260928 23:19:27.294095 15112 tablet_bootstrap.cc:654] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:27.294095 15113 tablet_bootstrap.cc:654] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:27.294628 15113 log.cc:824] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:27.294636 15112 log.cc:824] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:27.295323 15113 tablet_bootstrap.cc:492] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: No bootstrap required, opened a new log
I20260928 23:19:27.295392 15113 ts_tablet_manager.cc:1397] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:19:27.295511 15112 tablet_bootstrap.cc:492] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: No bootstrap required, opened a new log
I20260928 23:19:27.295595 15112 ts_tablet_manager.cc:1397] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:19:27.297286 15113 raft_consensus.cc:348] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.297473 15113 raft_consensus.cc:374] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:27.297389 15112 raft_consensus.cc:348] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.297518 15112 raft_consensus.cc:374] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:27.297514 15113 raft_consensus.cc:729] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f8bdf5850148a19c0245d6d073f5e2, State: Initialized, Role: FOLLOWER
I20260928 23:19:27.297555 15112 raft_consensus.cc:729] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a08d21578524f4686c468e80c414e11, State: Initialized, Role: FOLLOWER
I20260928 23:19:27.297631 15113 consensus_queue.cc:260] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.297645 15112 consensus_queue.cc:260] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.297899 15113 ts_tablet_manager.cc:1428] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:19:27.297899 15112 ts_tablet_manager.cc:1428] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:19:27.297910 15102 heartbeater.cc:503] Master 127.0.98.62:44711 was elected leader, sending a full tablet report...
I20260928 23:19:27.297981 14970 heartbeater.cc:503] Master 127.0.98.62:44711 was elected leader, sending a full tablet report...
I20260928 23:19:27.302157 14837 raft_consensus.cc:482] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:19:27.302245 14837 raft_consensus.cc:504] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.302516 14837 leader_election.cc:288] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523), e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279)
I20260928 23:19:27.305094 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.302785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48735 (local address 127.0.98.2:44279)
0928 23:19:27.302834 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.302836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.302843 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:19:27.302864 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.302865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.302874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.302924 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:27.303113 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.303566 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.304440 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.304805 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.304862 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.304870 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.305006 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.305037 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:19:27.305123 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.302684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48451 (local address 127.0.98.3:46523)
0928 23:19:27.302718 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.302720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.302860 (+ 140us) server_negotiation.cc:407] Connection header received
0928 23:19:27.302934 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.302935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.302942 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.302972 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:27.303100 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.303540 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.304462 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.304826 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.304881 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.304889 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.305036 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.305067 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:19:27.305130 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.302647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44279 (local address 127.0.98.1:48735)
0928 23:19:27.302695 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.302704 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:27.302748 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:27.303004 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:27.303004 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:27.303006 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:27.303054 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.303055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.303622 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.303622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.304362 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.304364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.304869 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.304870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.304936 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.304950 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:19:27.304994 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:19:27.305034 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 23:19:27.305248 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.302659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46523 (local address 127.0.98.1:48451)
0928 23:19:27.302799 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.302809 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:27.302871 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:27.302978 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:27.302978 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:27.302980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:27.303042 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.303044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.303625 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.303626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.304390 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:27.304392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.304914 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.304915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:27.304981 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.304983 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:19:27.305026 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:19:27.305064 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":8,"thread_start_us":81,"threads_started":1}
I20260928 23:19:27.305400 14924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91d5d043b914d4290d0c45548e8d071" candidate_uuid: "7c9e483323274aa1968278b79d1ec0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" is_pre_election: true
I20260928 23:19:27.305516 14924 raft_consensus.cc:2454] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c9e483323274aa1968278b79d1ec0f0 in term 0.
I20260928 23:19:27.305490 15056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91d5d043b914d4290d0c45548e8d071" candidate_uuid: "7c9e483323274aa1968278b79d1ec0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a08d21578524f4686c468e80c414e11" is_pre_election: true
I20260928 23:19:27.305594 15056 raft_consensus.cc:2454] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c9e483323274aa1968278b79d1ec0f0 in term 0.
I20260928 23:19:27.305703 14721 leader_election.cc:302] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c9e483323274aa1968278b79d1ec0f0, e4f8bdf5850148a19c0245d6d073f5e2; no voters:
I20260928 23:19:27.305846 14837 raft_consensus.cc:2781] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:19:27.305891 14837 raft_consensus.cc:482] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:19:27.305923 14837 raft_consensus.cc:3037] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:27.306576 14837 raft_consensus.cc:504] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.306703 14837 leader_election.cc:288] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523), e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279)
I20260928 23:19:27.306815 15056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91d5d043b914d4290d0c45548e8d071" candidate_uuid: "7c9e483323274aa1968278b79d1ec0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a08d21578524f4686c468e80c414e11"
I20260928 23:19:27.306883 15056 raft_consensus.cc:3037] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:27.306922 14924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91d5d043b914d4290d0c45548e8d071" candidate_uuid: "7c9e483323274aa1968278b79d1ec0f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f8bdf5850148a19c0245d6d073f5e2"
I20260928 23:19:27.307001 14924 raft_consensus.cc:3037] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:27.307483 15056 raft_consensus.cc:2454] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c9e483323274aa1968278b79d1ec0f0 in term 1.
I20260928 23:19:27.307607 14924 raft_consensus.cc:2454] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c9e483323274aa1968278b79d1ec0f0 in term 1.
I20260928 23:19:27.307655 14720 leader_election.cc:302] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a08d21578524f4686c468e80c414e11, 7c9e483323274aa1968278b79d1ec0f0; no voters:
I20260928 23:19:27.307732 14837 raft_consensus.cc:2781] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:19:27.307864 14837 raft_consensus.cc:686] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 LEADER]: Becoming Leader. State: Replica: 7c9e483323274aa1968278b79d1ec0f0, State: Running, Role: LEADER
I20260928 23:19:27.307933 14837 consensus_queue.cc:237] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.308575 14643 catalog_manager.cc:5495] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c9e483323274aa1968278b79d1ec0f0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7c9e483323274aa1968278b79d1ec0f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } health_report { overall_health: HEALTHY } } }
I20260928 23:19:27.316133 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.314032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.0.98.1:44309)
0928 23:19:27.314077 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:27.314079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:27.314086 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:19:27.314141 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:27.314142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:27.314157 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:27.314211 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:19:27.314309 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.314755 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.315421 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:27.315657 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:27.315711 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:27.315840 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:19:27.315946 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:19:27.316007 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:27.316051 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 23:19:27.316072 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:19:27.317314 15056 raft_consensus.cc:1261] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11 [term 1 FOLLOWER]: Refusing update from remote peer 7c9e483323274aa1968278b79d1ec0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:19:27.317315 14924 raft_consensus.cc:1261] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2 [term 1 FOLLOWER]: Refusing update from remote peer 7c9e483323274aa1968278b79d1ec0f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:19:27.317586 15119 consensus_queue.cc:1035] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:19:27.317696 14837 consensus_queue.cc:1035] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:19:27.319787 15121 mvcc.cc:205] Tried to move back new op lower bound from 7334451475729932288 to 7334451475693862912. Current Snapshot: MvccSnapshot[applied={T|T < 7334451475729932288}]
I20260928 23:19:27.319958 15122 mvcc.cc:205] Tried to move back new op lower bound from 7334451475729932288 to 7334451475693862912. Current Snapshot: MvccSnapshot[applied={T|T < 7334451475729932288}]
I20260928 23:19:27.321609 14644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:19:27.321679 14644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:19:27.322242 14644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Sending DeleteTablet for 3 replicas of tablet b91d5d043b914d4290d0c45548e8d071
I20260928 23:19:27.322523 14766 tablet_service.cc:1514] Processing DeleteTablet for tablet b91d5d043b914d4290d0c45548e8d071 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47622
I20260928 23:19:27.322538 14904 tablet_service.cc:1514] Processing DeleteTablet for tablet b91d5d043b914d4290d0c45548e8d071 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49628
I20260928 23:19:27.322551 15036 tablet_service.cc:1514] Processing DeleteTablet for tablet b91d5d043b914d4290d0c45548e8d071 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40898
I20260928 23:19:27.322602 14841 tablet_replica.cc:330] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: stopping tablet replica
I20260928 23:19:27.322642 14841 raft_consensus.cc:2229] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:19:27.322710 14841 raft_consensus.cc:2258] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:19:27.322835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 14683
I20260928 23:19:27.322961 15131 tablet_replica.cc:330] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: stopping tablet replica
I20260928 23:19:27.323020 14841 ts_tablet_manager.cc:1905] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:19:27.323127 15131 raft_consensus.cc:2229] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:19:27.323256 15131 raft_consensus.cc:2258] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:19:27.322738 15132 tablet_replica.cc:330] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: stopping tablet replica
I20260928 23:19:27.323519 15132 raft_consensus.cc:2229] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:19:27.323642 15132 raft_consensus.cc:2258] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:19:27.323697 15131 ts_tablet_manager.cc:1905] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:19:27.324254 15132 ts_tablet_manager.cc:1905] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:19:27.325222 15131 ts_tablet_manager.cc:1918] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:19:27.325285 15131 log.cc:1196] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/b91d5d043b914d4290d0c45548e8d071
I20260928 23:19:27.325553 15131 ts_tablet_manager.cc:1939] T b91d5d043b914d4290d0c45548e8d071 P e4f8bdf5850148a19c0245d6d073f5e2: Deleting consensus metadata
I20260928 23:19:27.325659 15132 ts_tablet_manager.cc:1918] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:19:27.325727 15132 log.cc:1196] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/b91d5d043b914d4290d0c45548e8d071
I20260928 23:19:27.325911 14631 catalog_manager.cc:4841] TS e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279): tablet b91d5d043b914d4290d0c45548e8d071 (table test-table [id=22c211130d5042c1b5c081ac2224c9b0]) successfully deleted
I20260928 23:19:27.326045 15132 ts_tablet_manager.cc:1939] T b91d5d043b914d4290d0c45548e8d071 P 4a08d21578524f4686c468e80c414e11: Deleting consensus metadata
I20260928 23:19:27.326372 14630 catalog_manager.cc:4841] TS 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523): tablet b91d5d043b914d4290d0c45548e8d071 (table test-table [id=22c211130d5042c1b5c081ac2224c9b0]) successfully deleted
W20260928 23:19:27.328194 14629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44309 (error 108)
I20260928 23:19:27.328593 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.328495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44309 (local address 127.0.0.1:47644)
0928 23:19:27.328538 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.328568 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:19:27.328691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 14838
W20260928 23:19:27.328701 14629 catalog_manager.cc:4584] TS 7c9e483323274aa1968278b79d1ec0f0 (127.0.98.1:44309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b91d5d043b914d4290d0c45548e8d071: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44309: connect: Connection refused (error 111)
I20260928 23:19:27.333356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 14973
I20260928 23:19:27.337854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 14611
I20260928 23:19:27.341933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44711
--webserver_interface=127.0.98.62
--webserver_port=35989
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:27.420392 15134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:27.420584 15134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:27.420614 15134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:27.421727 15134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:19:27.421765 15134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:27.421783 15134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:27.421799 15134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:19:27.421814 15134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:19:27.423388 15134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:27.423645 15134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:27.423885 15134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:27.428856 15139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:27.428866 15142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:27.428900 15134 server_base.cc:1034] running on GCE node
W20260928 23:19:27.428866 15140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:27.429306 15134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:27.429559 15134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:27.430692 15134 hybrid_clock.cc:648] HybridClock initialized: now 1790637567430684 us; error 26 us; skew 500 ppm
Sep 28 23:19:27 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:27.437516 15134 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:19:27.437954 15134 webserver.cc:460] Webserver started at http://127.0.98.62:35989/ using document root <none> and password file <none>
I20260928 23:19:27.438145 15134 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:27.438191 15134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:27.439424 15134 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:27.440093 15149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.440275 15134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.440362 15134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "898107bef00d473c842ff384f491c3b2"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "01225fa8211d30a90cbd0a7470ba0c08"
server_key_iv: "298a70ab06c5a7594bd861bbe5d1edba"
server_key_version: "encryptionkey@0"
I20260928 23:19:27.440697 15134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:27.448335 15134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:27.450253 15134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:27.450376 15134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:27.454056 15134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44711
I20260928 23:19:27.454089 15201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44711 every 8 connection(s)
I20260928 23:19:27.454401 15134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:19:27.455832 15202 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:19:27.455947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 15134
I20260928 23:19:27.456190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44309
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44711
--webserver_port=38129
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:19:27.456527 15202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Bootstrap starting.
I20260928 23:19:27.464080 15202 log.cc:824] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:27.466034 15202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:19:27.466264 15202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2: Bootstrap complete.
I20260928 23:19:27.468253 15202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:27.468470 15202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898107bef00d473c842ff384f491c3b2, State: Initialized, Role: FOLLOWER
I20260928 23:19:27.468611 15202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:27.468681 15202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:19:27.468729 15202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:19:27.468772 15202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 2 FOLLOWER]: Advancing to term 3
I20260928 23:19:27.469676 15202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:27.469765 15202 leader_election.cc:302] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 898107bef00d473c842ff384f491c3b2; no voters:
I20260928 23:19:27.469904 15202 leader_election.cc:288] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 23:19:27.469952 15206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 23:19:27.470062 15206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [term 3 LEADER]: Becoming Leader. State: Replica: 898107bef00d473c842ff384f491c3b2, State: Running, Role: LEADER
I20260928 23:19:27.470151 15202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:19:27.470140 15206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260928 23:19:27.470405 15206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 898107bef00d473c842ff384f491c3b2. Latest consensus state: current_term: 3 leader_uuid: "898107bef00d473c842ff384f491c3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } } }
I20260928 23:19:27.470472 15206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:27.470611 15207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "898107bef00d473c842ff384f491c3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898107bef00d473c842ff384f491c3b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } } }
I20260928 23:19:27.470739 15207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898107bef00d473c842ff384f491c3b2 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:27.470666 15210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:19:27.471601 15210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:19:27.471746 15210 catalog_manager.cc:1259] Loaded cluster ID: 2adeaec8dbb34980b3a4fbb59b0b927d
I20260928 23:19:27.471948 15210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 23:19:27.472183 15210 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc8c9a0e9ad google::LogMessage::Fail() at ??:0
@ 0x7fc8c9a1188a google::LogMessage::SendToLog() at ??:0
@ 0x7fc8c9a0e4de google::LogMessage::Flush() at ??:0
@ 0x7fc8c9a0fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc8d192f0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc8d19388b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc8cb8dc0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc8cb8d1954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc8cc7356db start_thread at ??:0
@ 0x7fc8c850c61f clone at ??:0
W20260928 23:19:27.539791 15204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:27.540000 15204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:27.540033 15204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:27.541139 15204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:27.541177 15204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:27.541219 15204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:19:27.542632 15204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38129
--tserver_master_addrs=127.0.98.62:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:27.542857 15204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:27.543051 15204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:27.547652 15238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:27.547820 15204 server_base.cc:1034] running on GCE node
W20260928 23:19:27.547662 15241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:27.547664 15239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:27.548149 15204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:27.548372 15204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:27.549527 15204 hybrid_clock.cc:648] HybridClock initialized: now 1790637567549511 us; error 31 us; skew 500 ppm
Sep 28 23:19:27 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:27.551888 15204 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:19:27.552321 15204 webserver.cc:460] Webserver started at http://127.0.98.1:38129/ using document root <none> and password file <none>
I20260928 23:19:27.552531 15204 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:27.552573 15204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:27.553793 15204 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:27.554535 15248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.554690 15204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:19:27.554807 15204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "7c9e483323274aa1968278b79d1ec0f0"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "8c905cefe2513e76b01d6d603a0647d7"
server_key_iv: "95eb17f9bdaa152a56d5b165155232be"
server_key_version: "encryptionkey@0"
I20260928 23:19:27.555096 15204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:27.577890 15204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:27.579851 15204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:27.579970 15204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:27.580171 15204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:27.580636 15255 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:19:27.582285 15204 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:19:27.582347 15204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 23:19:27.582382 15204 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:19:27.583132 15204 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:19:27.583175 15204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 23:19:27.583344 15255 tablet_bootstrap.cc:492] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Bootstrap starting.
I20260928 23:19:27.585853 15255 log.cc:824] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:27.588917 15255 tablet_bootstrap.cc:492] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:19:27.589278 15255 tablet_bootstrap.cc:492] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Bootstrap complete.
I20260928 23:19:27.589474 15255 ts_tablet_manager.cc:1397] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260928 23:19:27.589675 15204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44309
I20260928 23:19:27.590029 15204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:19:27.590598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 15204
I20260928 23:19:27.590843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44279
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44711
--webserver_port=37649
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:19:27.590586 15363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44309 every 8 connection(s)
I20260928 23:19:27.592805 15255 raft_consensus.cc:348] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.593286 15255 raft_consensus.cc:729] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9e483323274aa1968278b79d1ec0f0, State: Initialized, Role: FOLLOWER
I20260928 23:19:27.593475 15255 consensus_queue.cc:260] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:27.593781 15255 ts_tablet_manager.cc:1428] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
W20260928 23:19:27.657249 15366 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:19:27.590662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:44651)
0928 23:19:27.590875 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.590925 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:27.592998 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:27.657059 (+ 64061us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
W20260928 23:19:27.657449 15364 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44711 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44711: Client connection negotiation failed: client connection to 127.0.98.62:44711: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:19:27.657761 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.657658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:35107)
0928 23:19:27.657700 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.657740 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 23:19:27.658083 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.658010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:56673)
0928 23:19:27.658049 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.658069 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 23:19:27.658193 15364 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:19:27.670213 15367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:27.670399 15367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:27.670420 15367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:27.671494 15367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:27.671525 15367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:27.671583 15367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:19:27.673010 15367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37649
--tserver_master_addrs=127.0.98.62:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:27.673197 15367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:27.673377 15367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:27.677999 15374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:27.678051 15376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:27.678009 15373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:27.678145 15367 server_base.cc:1034] running on GCE node
I20260928 23:19:27.678375 15367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:27.678572 15367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:27.679695 15367 hybrid_clock.cc:648] HybridClock initialized: now 1790637567679682 us; error 29 us; skew 500 ppm
Sep 28 23:19:27 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:27.681947 15367 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:19:27.682343 15367 webserver.cc:460] Webserver started at http://127.0.98.2:37649/ using document root <none> and password file <none>
I20260928 23:19:27.682554 15367 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:27.682597 15367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:27.683799 15367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:19:27.684293 15383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.684434 15367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.684502 15367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "e4f8bdf5850148a19c0245d6d073f5e2"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "ffeb18cd2bcb79d783ecc05daa5389b5"
server_key_iv: "8b968ce5e91479684b5708735f3350f0"
server_key_version: "encryptionkey@0"
I20260928 23:19:27.684820 15367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:27.694612 15367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:27.696569 15367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:27.696688 15367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:27.696873 15367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:27.697134 15367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:27.697162 15367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.697182 15367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:27.697206 15367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.702548 15367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44279
I20260928 23:19:27.702615 15496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44279 every 8 connection(s)
I20260928 23:19:27.702890 15367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:19:27.703815 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.703458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:38031)
0928 23:19:27.703612 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.703677 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
W20260928 23:19:27.704152 15497 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44711 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44711: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
I20260928 23:19:27.704371 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.704293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:57177)
0928 23:19:27.704328 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.704353 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 23:19:27.704583 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.704524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:50553)
0928 23:19:27.704553 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.704571 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 23:19:27.704663 15497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:19:27.706005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 15367
I20260928 23:19:27.706290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46523
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44711
--webserver_port=44613
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:27.783342 15500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:27.783526 15500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:27.783571 15500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:27.784673 15500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:27.784709 15500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:27.784734 15500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:19:27.786116 15500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44613
--tserver_master_addrs=127.0.98.62:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:27.786283 15500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:27.786445 15500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:27.791129 15508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:27.791131 15505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:27.791167 15500 server_base.cc:1034] running on GCE node
W20260928 23:19:27.791162 15506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:27.791469 15500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:27.791702 15500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:27.792837 15500 hybrid_clock.cc:648] HybridClock initialized: now 1790637567792822 us; error 27 us; skew 500 ppm
Sep 28 23:19:27 dist-test-slave-djkr krb5kdc[14067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:27.795060 15500 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:19:27.795447 15500 webserver.cc:460] Webserver started at http://127.0.98.3:44613/ using document root <none> and password file <none>
I20260928 23:19:27.795663 15500 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:27.795711 15500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:27.796880 15500 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:27.797375 15515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.797523 15500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.797597 15500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "4a08d21578524f4686c468e80c414e11"
format_stamp: "Formatted at 2026-09-28 23:19:26 on dist-test-slave-djkr"
server_key: "1ba8f16900a9bcc1ea42509c8179ab83"
server_key_iv: "02d0b2ba2bdb7e0c8a514c08491afb05"
server_key_version: "encryptionkey@0"
I20260928 23:19:27.797897 15500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:27.810377 15500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:27.812357 15500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:27.812492 15500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:27.812688 15500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:27.812969 15500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:27.813001 15500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.813028 15500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:27.813050 15500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:27.818462 15500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46523
I20260928 23:19:27.818504 15628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46523 every 8 connection(s)
I20260928 23:19:27.818809 15500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:19:27.819746 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.819383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:48055)
0928 23:19:27.819520 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.819614 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
W20260928 23:19:27.820132 15629 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44711 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44711: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
I20260928 23:19:27.820384 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.820295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:44121)
0928 23:19:27.820334 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.820363 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 23:19:27.820616 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:27.820543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:38549)
0928 23:19:27.820570 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:27.820603 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 23:19:27.820700 15629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:19:27.820734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 15500
I20260928 23:19:28.658704 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:28.658498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:59205)
0928 23:19:28.658606 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:28.658642 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260928 23:19:28.705139 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:28.704941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:34745)
0928 23:19:28.705062 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:28.705095 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260928 23:19:28.821211 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:28.821000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:44351)
0928 23:19:28.821121 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:28.821157 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260928 23:19:29.395859 15636 raft_consensus.cc:482] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:19:29.395987 15636 raft_consensus.cc:504] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:29.396301 15636 leader_election.cc:288] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523), e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279)
Sep 28 23:19:29 dist-test-slave-djkr krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 28 23:19:29 dist-test-slave-djkr krb5kdc[14067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 23:19:29.407655 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:29.396463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44279 (local address 127.0.98.1:36363)
0928 23:19:29.396601 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:29.396611 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:29.396667 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:29.397558 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:29.397561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:29.398009 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:29.398497 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:29.398499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:29.399999 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:29.400000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:29.401224 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:29.401227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:29.402030 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:29.402031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:29.402173 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:29.402706 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:29.402716 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:29.404314 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:29.406139 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:29.406146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:29.406148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:29.406417 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:29.406651 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:29.406652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:29.406652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:29.406744 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:29.407113 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:29.407119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:29.407263 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 23:19:29.407366 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 23:19:29.407519 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":24,"thread_start_us":59,"threads_started":1}
I20260928 23:19:29.407684 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:29.396589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36363 (local address 127.0.98.2:44279)
0928 23:19:29.396742 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:29.396749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:29.396767 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:19:29.397289 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:29.397293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:29.397352 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:29.397510 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:29.398590 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:29.399884 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:29.401298 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:29.401943 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:29.402054 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:29.404423 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:29.404428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:29.404428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:29.404440 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:29.406064 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:29.406494 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:29.406496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:29.406496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:29.406589 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:29.406802 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:29.406803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:29.406804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:29.407048 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:29.407121 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:29.407422 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 23:19:29.407584 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260928 23:19:29.407812 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:29.396429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46523 (local address 127.0.98.1:36529)
0928 23:19:29.396555 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:29.396566 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:29.396626 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:29.397558 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:29.397561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:29.398022 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:29.398331 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:29.398339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:29.399982 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:29.399983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:29.401224 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:29.401227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:29.402028 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:29.402029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:29.402173 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:29.402747 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:29.402751 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:29.404727 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:29.406568 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:29.406569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:29.406570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:29.406746 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:29.406977 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:29.406985 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:29.406985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:29.407042 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:29.407419 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:29.407420 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:29.407485 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 23:19:29.407577 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 23:19:29.407662 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":78,"thread_start_us":47,"threads_started":1}
I20260928 23:19:29.408352 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:29.396547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36529 (local address 127.0.98.3:46523)
0928 23:19:29.396695 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:29.396707 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:29.396730 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:19:29.397290 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:29.397293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:29.397352 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:29.397505 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:29.398449 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:29.399884 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:29.401298 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:29.401943 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:29.402054 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:29.404848 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:29.404852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:29.404853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:29.404864 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:29.406502 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:29.406813 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:29.406815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:29.406816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:29.406919 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:29.407106 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:29.407106 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:29.407107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:29.407360 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:29.407430 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:29.407637 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 23:19:29.408261 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":54,"threads_started":1}
W20260928 23:19:29.408417 15252 leader_election.cc:341] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279): Not found: Tablet not found: b91d5d043b914d4290d0c45548e8d071
W20260928 23:19:29.408946 15251 leader_election.cc:341] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523): Not found: Tablet not found: b91d5d043b914d4290d0c45548e8d071
I20260928 23:19:29.409003 15251 leader_election.cc:302] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c9e483323274aa1968278b79d1ec0f0; no voters: 4a08d21578524f4686c468e80c414e11, e4f8bdf5850148a19c0245d6d073f5e2
I20260928 23:19:29.409094 15636 raft_consensus.cc:2726] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:19:29.659304 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:29.659179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:55755)
0928 23:19:29.659224 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:29.659262 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:19:29.705701 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:29.705494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:49517)
0928 23:19:29.705614 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:29.705658 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260928 23:19:29.821835 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:29.821626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:55007)
0928 23:19:29.821752 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:29.821795 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260928 23:19:30.659981 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:30.659707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:55291)
0928 23:19:30.659878 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:30.659916 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":59,"threads_started":1}
I20260928 23:19:30.706233 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:30.706054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:45639)
0928 23:19:30.706159 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:30.706191 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
I20260928 23:19:30.822466 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:30.822270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:37599)
0928 23:19:30.822393 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:30.822427 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260928 23:19:31.636349 15646 raft_consensus.cc:482] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:19:31.636446 15646 raft_consensus.cc:504] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:31.636634 15646 leader_election.cc:288] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523), e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279)
W20260928 23:19:31.637017 15251 leader_election.cc:341] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523): Not found: Tablet not found: b91d5d043b914d4290d0c45548e8d071
W20260928 23:19:31.637156 15252 leader_election.cc:341] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279): Not found: Tablet not found: b91d5d043b914d4290d0c45548e8d071
I20260928 23:19:31.637210 15252 leader_election.cc:302] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c9e483323274aa1968278b79d1ec0f0; no voters: 4a08d21578524f4686c468e80c414e11, e4f8bdf5850148a19c0245d6d073f5e2
I20260928 23:19:31.637287 15646 raft_consensus.cc:2726] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:19:31.660600 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:31.660404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:50537)
0928 23:19:31.660521 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:31.660556 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260928 23:19:31.706831 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:31.706622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:38319)
0928 23:19:31.706744 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:31.706780 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260928 23:19:31.823107 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:31.822904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:55405)
0928 23:19:31.823022 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:31.823056 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260928 23:19:32.661270 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:32.661065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:50889)
0928 23:19:32.661187 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:32.661223 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260928 23:19:32.707355 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:32.707184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:52681)
0928 23:19:32.707283 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:32.707314 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":55,"threads_started":1}
I20260928 23:19:32.823711 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:32.823499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:41645)
0928 23:19:32.823634 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:32.823667 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260928 23:19:33.661870 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:33.661659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:38833)
0928 23:19:33.661785 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:33.661821 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260928 23:19:33.707973 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:33.707746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:50833)
0928 23:19:33.707897 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:33.707934 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260928 23:19:33.824293 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:33.824070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:47733)
0928 23:19:33.824207 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:33.824253 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260928 23:19:34.662581 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:34.662320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:54165)
0928 23:19:34.662473 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:34.662520 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260928 23:19:34.708592 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:34.708374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:34029)
0928 23:19:34.708509 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:34.708545 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260928 23:19:34.824852 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:34.824630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:57851)
0928 23:19:34.824774 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:34.824812 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":58,"threads_started":1}
I20260928 23:19:34.860313 15659 raft_consensus.cc:482] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:19:34.860405 15659 raft_consensus.cc:504] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:34.860533 15659 leader_election.cc:288] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523), e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279)
W20260928 23:19:34.860894 15252 leader_election.cc:341] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279): Not found: Tablet not found: b91d5d043b914d4290d0c45548e8d071
W20260928 23:19:34.860997 15251 leader_election.cc:341] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523): Not found: Tablet not found: b91d5d043b914d4290d0c45548e8d071
I20260928 23:19:34.861042 15251 leader_election.cc:302] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c9e483323274aa1968278b79d1ec0f0; no voters: 4a08d21578524f4686c468e80c414e11, e4f8bdf5850148a19c0245d6d073f5e2
I20260928 23:19:34.861111 15659 raft_consensus.cc:2726] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:19:35.664173 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:35.663921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:51763)
0928 23:19:35.664075 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:35.664123 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260928 23:19:35.709245 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:35.709033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:45827)
0928 23:19:35.709154 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:35.709191 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260928 23:19:35.825433 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:35.825218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:59479)
0928 23:19:35.825354 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:35.825392 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260928 23:19:36.664872 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:36.664613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:38881)
0928 23:19:36.664755 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:36.664823 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260928 23:19:36.709848 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:36.709651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:52025)
0928 23:19:36.709772 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:36.709804 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260928 23:19:36.825984 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:36.825801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:43161)
0928 23:19:36.825904 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:36.825942 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260928 23:19:37.665496 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:37.665285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:40079)
0928 23:19:37.665414 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:37.665450 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":54,"threads_started":1}
I20260928 23:19:37.710474 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:37.710245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:35923)
0928 23:19:37.710392 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:37.710429 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":56,"threads_started":1}
I20260928 23:19:37.826610 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:37.826374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:59473)
0928 23:19:37.826499 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:37.826538 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260928 23:19:38.666218 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:38.665918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:54515)
0928 23:19:38.666125 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:38.666167 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260928 23:19:38.711097 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:38.710883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:45893)
0928 23:19:38.711018 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:38.711054 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260928 23:19:38.827265 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:38.827049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:57715)
0928 23:19:38.827184 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:38.827221 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260928 23:19:39.666878 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:39.666645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:47855)
0928 23:19:39.666793 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:39.666832 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260928 23:19:39.711786 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:39.711570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:49679)
0928 23:19:39.711706 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:39.711743 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260928 23:19:39.827924 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:39.827735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:40007)
0928 23:19:39.827845 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:39.827880 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260928 23:19:39.853657 15675 raft_consensus.cc:482] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:19:39.853749 15675 raft_consensus.cc:504] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a08d21578524f4686c468e80c414e11" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46523 } } peers { permanent_uuid: "e4f8bdf5850148a19c0245d6d073f5e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44279 } } peers { permanent_uuid: "7c9e483323274aa1968278b79d1ec0f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260928 23:19:39.853931 15675 leader_election.cc:288] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523), e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279)
W20260928 23:19:39.854267 15251 leader_election.cc:341] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a08d21578524f4686c468e80c414e11 (127.0.98.3:46523): Not found: Tablet not found: b91d5d043b914d4290d0c45548e8d071
W20260928 23:19:39.854355 15252 leader_election.cc:341] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e4f8bdf5850148a19c0245d6d073f5e2 (127.0.98.2:44279): Not found: Tablet not found: b91d5d043b914d4290d0c45548e8d071
I20260928 23:19:39.854416 15252 leader_election.cc:302] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c9e483323274aa1968278b79d1ec0f0; no voters: 4a08d21578524f4686c468e80c414e11, e4f8bdf5850148a19c0245d6d073f5e2
I20260928 23:19:39.854493 15675 raft_consensus.cc:2726] T b91d5d043b914d4290d0c45548e8d071 P 7c9e483323274aa1968278b79d1ec0f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:19:40.667594 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:40.667326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:48249)
0928 23:19:40.667469 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:40.667505 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260928 23:19:40.712404 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:40.712185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:55395)
0928 23:19:40.712320 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:40.712359 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260928 23:19:40.828517 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:40.828336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:44403)
0928 23:19:40.828441 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:40.828476 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260928 23:19:41.668231 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:41.668038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:34893)
0928 23:19:41.668154 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:41.668184 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260928 23:19:41.713016 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:41.712821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:57579)
0928 23:19:41.712935 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:41.712971 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260928 23:19:41.829145 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:41.828960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:48987)
0928 23:19:41.829069 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:41.829103 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260928 23:19:42.668926 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:42.668711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.1:44435)
0928 23:19:42.668828 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:42.668878 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260928 23:19:42.713737 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:42.713538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.2:50549)
0928 23:19:42.713659 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:42.713693 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260928 23:19:42.824188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 15204
I20260928 23:19:42.828392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 15367
I20260928 23:19:42.829762 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:42.829568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44711 (local address 127.0.98.3:39883)
0928 23:19:42.829696 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:42.829725 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260928 23:19:42.832087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 15500
2026-09-28T23:19:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17897 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:19:42 dist-test-slave-djkr krb5kdc[15687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:19:42 dist-test-slave-djkr krb5kdc[15687](info): set up 2 sockets
Sep 28 23:19:42 dist-test-slave-djkr krb5kdc[15687](info): commencing operation
krb5kdc: starting...
W20260928 23:19:43.783322 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:19:43 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:19:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:19:43.887574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:43.964464 15700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:43.964660 15700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:43.964680 15700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:43.965766 15700 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:19:43.965796 15700 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:19:43.965807 15700 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:19:43.965817 15700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:19:43.965829 15700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:43.965840 15700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:43.965857 15700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:19:43.965866 15700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:19:43.967290 15700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:43.967487 15700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:43.967679 15700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:43.972277 15705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:43.972301 15700 server_base.cc:1034] running on GCE node
W20260928 23:19:43.972249 15708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:43.972265 15706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:43.972640 15700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:43.972894 15700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:43.974041 15700 hybrid_clock.cc:648] HybridClock initialized: now 1790637583974028 us; error 29 us; skew 500 ppm
Sep 28 23:19:43 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:43.976249 15700 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:19:43.976609 15700 webserver.cc:460] Webserver started at http://127.0.98.62:46007/ using document root <none> and password file <none>
I20260928 23:19:43.976768 15700 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:43.976805 15700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:43.976900 15700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:43.977883 15700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "93aadaee7e254ccb8f1342ac914b2a50"
format_stamp: "Formatted at 2026-09-28 23:19:43 on dist-test-slave-djkr"
server_key: "b4931dfd9b34918fe51c8a11f08ee765"
server_key_iv: "4735543c2a4b46ae03dc07d47ed1278a"
server_key_version: "encryptionkey@0"
I20260928 23:19:43.978171 15700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "93aadaee7e254ccb8f1342ac914b2a50"
format_stamp: "Formatted at 2026-09-28 23:19:43 on dist-test-slave-djkr"
server_key: "b4931dfd9b34918fe51c8a11f08ee765"
server_key_iv: "4735543c2a4b46ae03dc07d47ed1278a"
server_key_version: "encryptionkey@0"
I20260928 23:19:43.979295 15700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:19:43.979975 15715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:43.980105 15700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:43.980182 15700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "93aadaee7e254ccb8f1342ac914b2a50"
format_stamp: "Formatted at 2026-09-28 23:19:43 on dist-test-slave-djkr"
server_key: "b4931dfd9b34918fe51c8a11f08ee765"
server_key_iv: "4735543c2a4b46ae03dc07d47ed1278a"
server_key_version: "encryptionkey@0"
I20260928 23:19:43.980240 15700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:43.989508 15700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:43.991204 15700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:43.991325 15700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:43.994874 15700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38957
I20260928 23:19:43.994913 15767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38957 every 8 connection(s)
I20260928 23:19:43.995203 15700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:19:43.996011 15768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:43.998122 15768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Bootstrap starting.
I20260928 23:19:43.998632 15768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:43.998864 15768 log.cc:824] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:43.999433 15768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: No bootstrap required, opened a new log
I20260928 23:19:44.000928 15768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:19:44.001061 15768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:44.001082 15768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93aadaee7e254ccb8f1342ac914b2a50, State: Initialized, Role: FOLLOWER
I20260928 23:19:44.001186 15768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:19:44.001257 15768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:19:44.001283 15768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:19:44.001322 15768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:44.001502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 15700
I20260928 23:19:44.001590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:19:44.001792 392 external_mini_cluster.cc:1428] Setting key 9eb937d7b11ebba5cf36a03bdaa4cd4f
I20260928 23:19:44.001943 15768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:19:44.002043 15768 leader_election.cc:302] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93aadaee7e254ccb8f1342ac914b2a50; no voters:
I20260928 23:19:44.002194 15768 leader_election.cc:288] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:19:44.002215 15771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:19:44.002382 15771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 1 LEADER]: Becoming Leader. State: Replica: 93aadaee7e254ccb8f1342ac914b2a50, State: Running, Role: LEADER
I20260928 23:19:44.002458 15771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:19:44.002722 15773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93aadaee7e254ccb8f1342ac914b2a50. Latest consensus state: current_term: 1 leader_uuid: "93aadaee7e254ccb8f1342ac914b2a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } } }
I20260928 23:19:44.002781 15771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93aadaee7e254ccb8f1342ac914b2a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } } }
I20260928 23:19:44.002799 15773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:44.002827 15771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: This master's current role is: LEADER
I20260928 23:19:44.002998 15768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:19:44.003070 15776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:19:44.003683 15776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:19:44.005165 15776 catalog_manager.cc:1347] Generated new cluster ID: f7fa62b6e77549fd81eacff50c421c2e
I20260928 23:19:44.005215 15776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:19:44 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:44.011860 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.002239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.0.98.62:38957)
0928 23:19:44.002711 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.002721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.002753 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:19:44.002875 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.002883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.002941 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.003857 (+ 916us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:44.003981 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.005318 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.006044 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.006452 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.006543 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.008689 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:44.008694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:44.008694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:44.008705 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:44.010508 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:44.010812 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:44.010815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:44.010815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:44.010902 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:44.011067 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:44.011068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:44.011069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:44.011284 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:44.011423 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.011512 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.011664 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":397,"threads_started":1}
I20260928 23:19:44.014896 15776 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:19:44.015355 15776 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:19:44.021538 15776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Generated new TSK 0
I20260928 23:19:44.021682 15776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:19:44.046943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38957
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:44.124809 15793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:44.125000 15793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:44.125021 15793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:44.126103 15793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:44.126137 15793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:44.126161 15793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:19:44.127516 15793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:44.127753 15793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:44.127954 15793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:19:44.132648 15793 server_base.cc:1034] running on GCE node
W20260928 23:19:44.132680 15799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:44.132649 15798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:44.132755 15801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:44.133122 15793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:44.133327 15793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:44.134469 15793 hybrid_clock.cc:648] HybridClock initialized: now 1790637584134454 us; error 30 us; skew 500 ppm
Sep 28 23:19:44 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:44.137102 15793 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:19:44.137534 15793 webserver.cc:460] Webserver started at http://127.0.98.1:34001/ using document root <none> and password file <none>
I20260928 23:19:44.137722 15793 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:44.137768 15793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:44.137889 15793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:44.138957 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "ab568863eb584ef594447579a0aec55a"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "8f6b55a8ac2c4605fbd8d38d48b972c8"
server_key_iv: "d44d7b0ac84363808487467312d5acfd"
server_key_version: "encryptionkey@0"
I20260928 23:19:44.139281 15793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ab568863eb584ef594447579a0aec55a"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "8f6b55a8ac2c4605fbd8d38d48b972c8"
server_key_iv: "d44d7b0ac84363808487467312d5acfd"
server_key_version: "encryptionkey@0"
I20260928 23:19:44.140530 15793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:19:44.141234 15808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:44.141398 15793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:19:44.141475 15793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "ab568863eb584ef594447579a0aec55a"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "8f6b55a8ac2c4605fbd8d38d48b972c8"
server_key_iv: "d44d7b0ac84363808487467312d5acfd"
server_key_version: "encryptionkey@0"
I20260928 23:19:44.141542 15793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:44.150317 15793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:44.152113 15793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:44.152242 15793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:44.152447 15793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:44.152714 15793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:44.152745 15793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:44.152772 15793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:44.152789 15793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:44.158109 15793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45869
I20260928 23:19:44.158177 15921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45869 every 8 connection(s)
I20260928 23:19:44.158463 15793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:19:44.161345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 15793
I20260928 23:19:44.161439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:19:44.161620 392 external_mini_cluster.cc:1428] Setting key a5417f8286066c2fd1f2f9a7629358e2
Sep 28 23:19:44 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:44.167389 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.159134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33583 (local address 127.0.98.62:38957)
0928 23:19:44.159208 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.159211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.159397 (+ 186us) server_negotiation.cc:407] Connection header received
0928 23:19:44.159940 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.159941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.159952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.159993 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:44.160890 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.161422 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.162581 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.162780 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.162834 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.164954 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:44.164956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:44.164956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:44.164964 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:44.166193 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:44.166546 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:44.166547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:44.166548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:44.166598 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:44.166813 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:44.166814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:44.166814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:44.166913 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:44.166968 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.167227 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.167303 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 23:19:44.167567 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.159070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:33583)
0928 23:19:44.159275 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:44.159319 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:44.159839 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:44.160048 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:44.160056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:44.160593 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:44.160791 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.160796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.161552 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.161553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.162510 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.162513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.162848 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.162849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.162974 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.163439 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:44.163447 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:44.164866 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:44.166269 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:44.166276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:44.166278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:44.166484 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:44.166652 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:44.166653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:44.166653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:44.166762 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:44.166967 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:44.166974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:44.167104 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 23:19:44.167187 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:19:44.167337 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
I20260928 23:19:44.168130 15922 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38957
I20260928 23:19:44.168222 15922 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:44.168380 15922 heartbeater.cc:511] Master 127.0.98.62:38957 requested a full tablet report, sending...
I20260928 23:19:44.168905 15732 ts_manager.cc:194] Registered new tserver with Master: ab568863eb584ef594447579a0aec55a (127.0.98.1:45869)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:19:44.169719 15732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:19:44.183857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38957
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:44.261015 15927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:44.261209 15927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:44.261238 15927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:44.262341 15927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:44.262389 15927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:44.262423 15927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:19:44.263828 15927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:44.264045 15927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:44.264245 15927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:44.268792 15935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:44.268817 15932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:44.268833 15933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:44.268962 15927 server_base.cc:1034] running on GCE node
I20260928 23:19:44.269187 15927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:44.269409 15927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:44.270547 15927 hybrid_clock.cc:648] HybridClock initialized: now 1790637584270531 us; error 33 us; skew 500 ppm
Sep 28 23:19:44 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:44.272794 15927 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:19:44.273204 15927 webserver.cc:460] Webserver started at http://127.0.98.2:42267/ using document root <none> and password file <none>
I20260928 23:19:44.273404 15927 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:44.273449 15927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:44.273564 15927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:44.274520 15927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "2321b542914c4e9484e45de0c2f5d781"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "11b1a29c4e3193b52b5bd82acb4b3562"
server_key_iv: "4628f6dbb200c755213653d25cd32844"
server_key_version: "encryptionkey@0"
I20260928 23:19:44.274827 15927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2321b542914c4e9484e45de0c2f5d781"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "11b1a29c4e3193b52b5bd82acb4b3562"
server_key_iv: "4628f6dbb200c755213653d25cd32844"
server_key_version: "encryptionkey@0"
I20260928 23:19:44.276022 15927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:19:44.276638 15942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:44.276777 15927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:19:44.276844 15927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "2321b542914c4e9484e45de0c2f5d781"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "11b1a29c4e3193b52b5bd82acb4b3562"
server_key_iv: "4628f6dbb200c755213653d25cd32844"
server_key_version: "encryptionkey@0"
I20260928 23:19:44.276907 15927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:44.282969 15927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:44.284611 15927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:44.284742 15927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:44.284932 15927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:44.285181 15927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:44.285211 15927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:44.285238 15927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:44.285261 15927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:44.290645 15927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39533
I20260928 23:19:44.290711 16055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39533 every 8 connection(s)
I20260928 23:19:44.290989 15927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:19:44 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:44.297530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 15927
I20260928 23:19:44.297623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:19:44.297803 392 external_mini_cluster.cc:1428] Setting key 3b9b88b6641bb99f0171f200e1611f48
I20260928 23:19:44.299352 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.291659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53845 (local address 127.0.98.62:38957)
0928 23:19:44.291717 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.291719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.291891 (+ 172us) server_negotiation.cc:407] Connection header received
0928 23:19:44.292356 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.292357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.292367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.292415 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:44.293289 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.293787 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.294747 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.294947 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.295008 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.296880 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:44.296881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:44.296882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:44.296888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:44.298160 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:44.298505 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:44.298506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:44.298506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:44.298561 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:44.298763 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:44.298763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:44.298764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:44.298881 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:44.298940 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.299178 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.299256 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:19:44.299506 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.291659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:53845)
0928 23:19:44.291787 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:44.291821 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:44.292273 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:44.292471 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:44.292474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:44.293004 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:44.293187 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.293191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.293855 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.293855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.294656 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.294657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.295012 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.295012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.295099 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.295521 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:44.295529 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:44.296801 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:44.298226 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:44.298233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:44.298235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:44.298440 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:44.298615 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:44.298615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:44.298616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:44.298709 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:44.298937 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:44.298943 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:44.299069 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 23:19:44.299152 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:19:44.299299 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
I20260928 23:19:44.300029 16056 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38957
I20260928 23:19:44.300127 16056 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:44.300304 16056 heartbeater.cc:511] Master 127.0.98.62:38957 requested a full tablet report, sending...
I20260928 23:19:44.300652 15732 ts_manager.cc:194] Registered new tserver with Master: 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533)
I20260928 23:19:44.301009 15732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53845
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:19:44.320483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38957
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:19:44.397377 16061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:19:44.397562 16061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:19:44.397581 16061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:19:44.398723 16061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:19:44.398766 16061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:19:44.398792 16061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:19:44.400236 16061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:19:44.400445 16061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:19:44.400640 16061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:19:44.405272 16066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:44.405441 16061 server_base.cc:1034] running on GCE node
W20260928 23:19:44.405280 16069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:19:44.405316 16067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:19:44.405776 16061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:19:44.405987 16061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:19:44.407133 16061 hybrid_clock.cc:648] HybridClock initialized: now 1790637584407116 us; error 34 us; skew 500 ppm
Sep 28 23:19:44 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:19:44.409500 16061 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:19:44.409904 16061 webserver.cc:460] Webserver started at http://127.0.98.3:42111/ using document root <none> and password file <none>
I20260928 23:19:44.410104 16061 fs_manager.cc:359] Metadata directory not provided
I20260928 23:19:44.410151 16061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:19:44.410272 16061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:19:44.411263 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "12d68848d62c49c19db5569ad31ccaf9"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "00f8033ac55a03e7a6e0375fb60e0339"
server_key_iv: "6c04d1ef622c20318984bd41e8a6779d"
server_key_version: "encryptionkey@0"
I20260928 23:19:44.411618 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "12d68848d62c49c19db5569ad31ccaf9"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "00f8033ac55a03e7a6e0375fb60e0339"
server_key_iv: "6c04d1ef622c20318984bd41e8a6779d"
server_key_version: "encryptionkey@0"
I20260928 23:19:44.412830 16061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:19:44.413446 16076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:44.413589 16061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:19:44.413668 16061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "12d68848d62c49c19db5569ad31ccaf9"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "00f8033ac55a03e7a6e0375fb60e0339"
server_key_iv: "6c04d1ef622c20318984bd41e8a6779d"
server_key_version: "encryptionkey@0"
I20260928 23:19:44.413731 16061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:19:44.428187 16061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:19:44.429842 16061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:19:44.429965 16061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:19:44.430146 16061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:19:44.430418 16061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:19:44.430449 16061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:44.430469 16061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:19:44.430491 16061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:19:44.435775 16061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45153
I20260928 23:19:44.435838 16189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45153 every 8 connection(s)
I20260928 23:19:44.436095 16061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:19:44 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:19:44.444065 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.436725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37787 (local address 127.0.98.62:38957)
0928 23:19:44.436805 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.436807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.436954 (+ 147us) server_negotiation.cc:407] Connection header received
0928 23:19:44.437400 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.437401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.437411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.437454 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:44.438319 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.438887 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.439892 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.440068 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.440111 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.441934 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:44.441936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:44.441936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:44.441943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:44.442990 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:44.443308 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:44.443309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:44.443309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:44.443351 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:44.443582 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:44.443583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:44.443583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:44.443685 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:44.443735 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.443941 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.444002 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:19:44.444211 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.436694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:37787)
0928 23:19:44.436842 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:44.436877 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:44.437326 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:44.437512 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:44.437515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:44.438066 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:19:44.438247 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.438253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.438970 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.438971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.439831 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.439834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.440117 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.440118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.440207 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.440582 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:19:44.440589 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:19:44.441855 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:19:44.443052 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:44.443057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:44.443059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:44.443254 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:44.443400 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:19:44.443401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:19:44.443401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:19:44.443510 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:19:44.443731 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:19:44.443734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:19:44.443846 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 23:19:44.443923 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:19:44.444034 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260928 23:19:44.444525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 16061
I20260928 23:19:44.444630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:19:44.444659 16190 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38957
I20260928 23:19:44.444788 16190 heartbeater.cc:464] Registering TS with master...
I20260928 23:19:44.444835 392 external_mini_cluster.cc:1428] Setting key 2ad22910ef7029cd8cca1d759c242913
I20260928 23:19:44.444960 16190 heartbeater.cc:511] Master 127.0.98.62:38957 requested a full tablet report, sending...
I20260928 23:19:44.445303 15732 ts_manager.cc:194] Registered new tserver with Master: 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153)
I20260928 23:19:44.445659 15732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37787
I20260928 23:19:44.445674 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:19:44.452267 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.446914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.0.98.62:38957)
0928 23:19:44.446964 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.446965 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.447049 (+ 84us) server_negotiation.cc:407] Connection header received
0928 23:19:44.447092 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.447092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.447102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.447167 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:19:44.447694 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.448200 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.448891 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.449090 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.449153 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.450206 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:19:44.450207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:19:44.450207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:19:44.450213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:19:44.451401 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:44.451698 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:44.451699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:44.451699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:44.451744 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:19:44.451900 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:19:44.451900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:19:44.451900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:19:44.452024 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:19:44.452087 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.452152 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.452204 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:19:44.453799 15732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:19:44.455284 15732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:19:44.461186 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.457193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45153 (local address 127.0.0.1:60744)
0928 23:19:44.457519 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:44.457562 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:44.457642 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:44.457832 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:44.457834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:44.457836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:44.457887 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.457888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.459223 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.459224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.460205 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.460208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.460952 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.460953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.461026 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.461040 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:19:44.461098 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 23:19:44.461135 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":31,"thread_start_us":76,"threads_started":1}
I20260928 23:19:44.461243 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.457262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60744 (local address 127.0.98.3:45153)
0928 23:19:44.457441 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.457446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.457641 (+ 195us) server_negotiation.cc:407] Connection header received
0928 23:19:44.457734 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.457736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.457753 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.457829 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:44.458015 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.459113 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.460273 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.460865 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.460933 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.460987 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.461121 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.461161 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260928 23:19:44.461542 16198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.457193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39533 (local address 127.0.0.1:52388)
0928 23:19:44.457580 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:44.457589 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:44.457646 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:44.458084 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:44.458084 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:44.458086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:44.458128 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.458129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.458997 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.458998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.460198 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.460201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.461350 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.461350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.461407 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.461410 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:19:44.461451 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 23:19:44.461486 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":45,"thread_start_us":68,"threads_started":1}
I20260928 23:19:44.461638 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.457397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45869 (local address 127.0.0.1:46472)
0928 23:19:44.457773 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:44.457782 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:44.457818 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:44.458151 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:44.458151 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:44.458153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:44.458180 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.458181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.459407 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.459408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.460508 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.460510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.461418 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.461419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.461480 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.461484 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:19:44.461531 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:19:44.461572 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":60,"threads_started":1}
I20260928 23:19:44.461809 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.457336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.0.98.2:39533)
0928 23:19:44.457489 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.457494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.457877 (+ 383us) server_negotiation.cc:407] Connection header received
0928 23:19:44.457924 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.457927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.457942 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.458007 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:44.458197 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.458902 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.460299 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.460873 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.460940 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.460995 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.461539 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.461748 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260928 23:19:44.461894 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.457601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.0.98.1:45869)
0928 23:19:44.457958 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.457964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.457972 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:19:44.458005 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.458007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.458020 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.458082 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:44.458248 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.459309 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.460585 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.461262 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.461326 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.461382 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.461769 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.461814 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":104,"threads_started":1}
I20260928 23:19:44.461944 16124 tablet_service.cc:1467] Processing CreateTablet for tablet 14bd46162a254d0ca631bd9e7588301d (DEFAULT_TABLE table=test-table [id=3d2e2d016a2b4e5fba37e1c369aeed4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:44.462335 16124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bd46162a254d0ca631bd9e7588301d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:44.462422 15990 tablet_service.cc:1467] Processing CreateTablet for tablet 14bd46162a254d0ca631bd9e7588301d (DEFAULT_TABLE table=test-table [id=3d2e2d016a2b4e5fba37e1c369aeed4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:44.462715 15990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bd46162a254d0ca631bd9e7588301d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:44.465049 16204 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: Bootstrap starting.
I20260928 23:19:44.465373 15856 tablet_service.cc:1467] Processing CreateTablet for tablet 14bd46162a254d0ca631bd9e7588301d (DEFAULT_TABLE table=test-table [id=3d2e2d016a2b4e5fba37e1c369aeed4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:19:44.465728 15856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bd46162a254d0ca631bd9e7588301d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:19:44.465782 16204 tablet_bootstrap.cc:654] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:44.466280 16204 log.cc:824] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:44.467036 16204 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: No bootstrap required, opened a new log
I20260928 23:19:44.467106 16204 ts_tablet_manager.cc:1397] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:19:44.467696 16205 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Bootstrap starting.
I20260928 23:19:44.468219 16205 tablet_bootstrap.cc:654] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:44.468725 16205 log.cc:824] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:44.468907 16207 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Bootstrap starting.
I20260928 23:19:44.468993 16204 raft_consensus.cc:348] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:19:44.469151 16204 raft_consensus.cc:374] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:44.469182 16204 raft_consensus.cc:729] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d68848d62c49c19db5569ad31ccaf9, State: Initialized, Role: FOLLOWER
I20260928 23:19:44.469292 16204 consensus_queue.cc:260] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:19:44.469477 16207 tablet_bootstrap.cc:654] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Neither blocks nor log segments found. Creating new log.
I20260928 23:19:44.469558 16204 ts_tablet_manager.cc:1428] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:19:44.469565 16190 heartbeater.cc:503] Master 127.0.98.62:38957 was elected leader, sending a full tablet report...
I20260928 23:19:44.470132 16205 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: No bootstrap required, opened a new log
I20260928 23:19:44.470184 16205 ts_tablet_manager.cc:1397] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:19:44.470315 16207 log.cc:824] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:19:44.470914 16207 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: No bootstrap required, opened a new log
I20260928 23:19:44.470971 16207 ts_tablet_manager.cc:1397] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:19:44.471773 16205 raft_consensus.cc:348] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:19:44.471905 16205 raft_consensus.cc:374] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:44.471925 16205 raft_consensus.cc:729] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2321b542914c4e9484e45de0c2f5d781, State: Initialized, Role: FOLLOWER
I20260928 23:19:44.472024 16205 consensus_queue.cc:260] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:19:44.472271 16205 ts_tablet_manager.cc:1428] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:19:44.472309 16056 heartbeater.cc:503] Master 127.0.98.62:38957 was elected leader, sending a full tablet report...
I20260928 23:19:44.472474 16207 raft_consensus.cc:348] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:19:44.472661 16207 raft_consensus.cc:374] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:19:44.472699 16207 raft_consensus.cc:729] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab568863eb584ef594447579a0aec55a, State: Initialized, Role: FOLLOWER
I20260928 23:19:44.472800 16207 consensus_queue.cc:260] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:19:44.473065 16207 ts_tablet_manager.cc:1428] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:19:44.473109 15922 heartbeater.cc:503] Master 127.0.98.62:38957 was elected leader, sending a full tablet report...
I20260928 23:19:44.527125 16212 raft_consensus.cc:482] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:19:44.527246 16212 raft_consensus.cc:504] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:19:44.527596 16212 leader_election.cc:288] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533), 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153)
I20260928 23:19:44.530442 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.527806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55961 (local address 127.0.98.2:39533)
0928 23:19:44.527846 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.527847 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.527986 (+ 139us) server_negotiation.cc:407] Connection header received
0928 23:19:44.528033 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.528034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.528046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.528089 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:44.528246 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.528696 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.529787 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.530144 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.530204 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.530214 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.530340 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.530384 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:19:44.530449 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.527728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39533 (local address 127.0.98.1:55961)
0928 23:19:44.527927 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:44.527935 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:44.527980 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:44.528095 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:44.528096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:44.528097 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:44.528186 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.528188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.528762 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.528763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.529704 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.529707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.530211 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.530212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.530284 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.530287 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:19:44.530331 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:19:44.530368 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":14,"thread_start_us":113,"threads_started":1}
I20260928 23:19:44.530491 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.527850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43157 (local address 127.0.98.3:45153)
0928 23:19:44.527904 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.527906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.527916 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:19:44.527942 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.527943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.527951 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.528003 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:19:44.528151 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.528689 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.529833 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.530186 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.530245 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.530254 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.530397 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.530431 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:19:44.530539 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.527728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45153 (local address 127.0.98.1:43157)
0928 23:19:44.527791 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:44.527803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:19:44.527859 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:19:44.528021 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:19:44.528022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:19:44.528024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:19:44.528093 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.528095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.528758 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.528759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.529729 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:19:44.529731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.530257 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.530258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:19:44.530343 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.530345 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:19:44.530387 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:19:44.530423 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 23:19:44.530876 16010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd46162a254d0ca631bd9e7588301d" candidate_uuid: "ab568863eb584ef594447579a0aec55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2321b542914c4e9484e45de0c2f5d781" is_pre_election: true
I20260928 23:19:44.530936 16144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd46162a254d0ca631bd9e7588301d" candidate_uuid: "ab568863eb584ef594447579a0aec55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d68848d62c49c19db5569ad31ccaf9" is_pre_election: true
I20260928 23:19:44.531004 16010 raft_consensus.cc:2454] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab568863eb584ef594447579a0aec55a in term 0.
I20260928 23:19:44.531085 16144 raft_consensus.cc:2454] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab568863eb584ef594447579a0aec55a in term 0.
I20260928 23:19:44.531211 15811 leader_election.cc:302] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2321b542914c4e9484e45de0c2f5d781, ab568863eb584ef594447579a0aec55a; no voters:
I20260928 23:19:44.531375 16212 raft_consensus.cc:2781] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:19:44.531414 16212 raft_consensus.cc:482] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:19:44.531435 16212 raft_consensus.cc:3037] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:44.532115 16212 raft_consensus.cc:504] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:19:44.532227 16212 leader_election.cc:288] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 1 election: Requested vote from peers 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533), 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153)
I20260928 23:19:44.532368 16144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd46162a254d0ca631bd9e7588301d" candidate_uuid: "ab568863eb584ef594447579a0aec55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d68848d62c49c19db5569ad31ccaf9"
I20260928 23:19:44.532429 16144 raft_consensus.cc:3037] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:44.532464 16010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bd46162a254d0ca631bd9e7588301d" candidate_uuid: "ab568863eb584ef594447579a0aec55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2321b542914c4e9484e45de0c2f5d781"
I20260928 23:19:44.532545 16010 raft_consensus.cc:3037] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:19:44.533097 16144 raft_consensus.cc:2454] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab568863eb584ef594447579a0aec55a in term 1.
I20260928 23:19:44.533255 15810 leader_election.cc:302] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d68848d62c49c19db5569ad31ccaf9, ab568863eb584ef594447579a0aec55a; no voters:
I20260928 23:19:44.533268 16010 raft_consensus.cc:2454] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab568863eb584ef594447579a0aec55a in term 1.
I20260928 23:19:44.533334 16212 raft_consensus.cc:2781] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:19:44.533455 16212 raft_consensus.cc:686] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 1 LEADER]: Becoming Leader. State: Replica: ab568863eb584ef594447579a0aec55a, State: Running, Role: LEADER
I20260928 23:19:44.533540 16212 consensus_queue.cc:237] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:19:44.534278 15731 catalog_manager.cc:5495] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a reported cstate change: term changed from 0 to 1, leader changed from <none> to ab568863eb584ef594447579a0aec55a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ab568863eb584ef594447579a0aec55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } health_report { overall_health: UNKNOWN } } }
W20260928 23:19:44.541466 16057 tablet.cc:2369] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:19:44.551363 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.549327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.0.98.1:45869)
0928 23:19:44.549373 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:19:44.549374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:19:44.549382 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:19:44.549412 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:19:44.549413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:19:44.549428 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:19:44.549478 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:19:44.549578 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.550064 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.550750 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:19:44.550942 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:19:44.550985 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:19:44.551102 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:19:44.551194 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:19:44.551251 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:19:44.551270 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 23:19:44.551285 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:19:44.552984 16144 raft_consensus.cc:1261] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9 [term 1 FOLLOWER]: Refusing update from remote peer ab568863eb584ef594447579a0aec55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:19:44.553017 16010 raft_consensus.cc:1261] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Refusing update from remote peer ab568863eb584ef594447579a0aec55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:19:44.553294 16212 consensus_queue.cc:1035] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:19:44.553426 16214 consensus_queue.cc:1035] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:19:44.555737 16218 mvcc.cc:205] Tried to move back new op lower bound from 7334451546326671360 to 7334451546249879552. Current Snapshot: MvccSnapshot[applied={T|T < 7334451546326671360}]
I20260928 23:19:44.557869 15732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:19:44.557945 15732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:19:44.558549 15732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Sending DeleteTablet for 3 replicas of tablet 14bd46162a254d0ca631bd9e7588301d
I20260928 23:19:44.558876 15990 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bd46162a254d0ca631bd9e7588301d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52388
I20260928 23:19:44.558923 16124 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bd46162a254d0ca631bd9e7588301d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60744
I20260928 23:19:44.558928 15856 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bd46162a254d0ca631bd9e7588301d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46472
I20260928 23:19:44.559128 16228 tablet_replica.cc:330] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: stopping tablet replica
I20260928 23:19:44.559178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 15793
I20260928 23:19:44.559262 16228 raft_consensus.cc:2229] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:19:44.559368 16228 raft_consensus.cc:2258] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:19:44.560094 16228 ts_tablet_manager.cc:1905] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:19:44.561460 16228 ts_tablet_manager.cc:1918] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:19:44.561518 16228 log.cc:1196] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/14bd46162a254d0ca631bd9e7588301d
I20260928 23:19:44.561738 16228 ts_tablet_manager.cc:1939] T 14bd46162a254d0ca631bd9e7588301d P 12d68848d62c49c19db5569ad31ccaf9: Deleting consensus metadata
I20260928 23:19:44.562104 15717 catalog_manager.cc:4841] TS 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153): tablet 14bd46162a254d0ca631bd9e7588301d (table test-table [id=3d2e2d016a2b4e5fba37e1c369aeed4b]) successfully deleted
I20260928 23:19:44.563653 16226 tablet_replica.cc:330] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: stopping tablet replica
I20260928 23:19:44.563745 16226 raft_consensus.cc:2229] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:19:44.563841 16226 raft_consensus.cc:2258] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 23:19:44.564244 15718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45869 (error 108)
I20260928 23:19:44.564357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 15927
I20260928 23:19:44.565716 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.565604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45869 (local address 127.0.0.1:46488)
0928 23:19:44.565653 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:44.565689 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 23:19:44.565831 15718 catalog_manager.cc:4584] TS ab568863eb584ef594447579a0aec55a (127.0.98.1:45869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14bd46162a254d0ca631bd9e7588301d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45869: connect: Connection refused (error 111)
I20260928 23:19:44.568684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 16061
I20260928 23:19:44.568809 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:19:44.568705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39533 (local address 127.0.0.1:52400)
0928 23:19:44.568751 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 23:19:44.568782 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:19:44.572595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 15700
I20260928 23:20:04.576901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38957
--webserver_interface=127.0.98.62
--webserver_port=46007
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:04.655848 16230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:04.656033 16230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:04.656054 16230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:04.657143 16230 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:20:04.657181 16230 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:20:04.657192 16230 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:20:04.657202 16230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:04.657214 16230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:04.657229 16230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:04.657262 16230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:04.657284 16230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:04.658707 16230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:04.658917 16230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:04.659121 16230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:04.671948 16239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:04.672102 16230 server_base.cc:1034] running on GCE node
W20260928 23:20:04.672027 16237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:04.671953 16236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:04.672502 16230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:04.672820 16230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:04.673967 16230 hybrid_clock.cc:648] HybridClock initialized: now 1790637604673955 us; error 28 us; skew 500 ppm
Sep 28 23:20:04 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:04.676355 16230 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:20:04.676782 16230 webserver.cc:460] Webserver started at http://127.0.98.62:46007/ using document root <none> and password file <none>
I20260928 23:20:04.677001 16230 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:04.677062 16230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:04.678421 16230 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:04.679000 16246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:04.679149 16230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:04.679224 16230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "93aadaee7e254ccb8f1342ac914b2a50"
format_stamp: "Formatted at 2026-09-28 23:19:43 on dist-test-slave-djkr"
server_key: "b4931dfd9b34918fe51c8a11f08ee765"
server_key_iv: "4735543c2a4b46ae03dc07d47ed1278a"
server_key_version: "encryptionkey@0"
I20260928 23:20:04.679641 16230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:04.690928 16230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:04.692947 16230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:04.693074 16230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:04.696619 16230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38957
I20260928 23:20:04.696677 16298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38957 every 8 connection(s)
I20260928 23:20:04.697002 16230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:04.698411 16299 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:20:04.699034 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Bootstrap starting.
I20260928 23:20:04.701647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 16230
I20260928 23:20:04.701941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45869
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38957
--webserver_port=34001
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:20:04.707525 16299 log.cc:824] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:04.709291 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:20:04.709534 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Bootstrap complete.
I20260928 23:20:04.711627 16299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:20:04.711877 16299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93aadaee7e254ccb8f1342ac914b2a50, State: Initialized, Role: FOLLOWER
I20260928 23:20:04.712037 16299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:20:04.712108 16299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:04.712160 16299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:04.712205 16299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:20:04.713179 16299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:20:04.713274 16299 leader_election.cc:302] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93aadaee7e254ccb8f1342ac914b2a50; no voters:
I20260928 23:20:04.713425 16299 leader_election.cc:288] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:20:04.713482 16303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:20:04.713662 16299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:04.713685 16303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 2 LEADER]: Becoming Leader. State: Replica: 93aadaee7e254ccb8f1342ac914b2a50, State: Running, Role: LEADER
I20260928 23:20:04.713781 16303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:20:04.714017 16303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93aadaee7e254ccb8f1342ac914b2a50. Latest consensus state: current_term: 2 leader_uuid: "93aadaee7e254ccb8f1342ac914b2a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } } }
I20260928 23:20:04.714041 16304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "93aadaee7e254ccb8f1342ac914b2a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } } }
I20260928 23:20:04.714159 16304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:04.714336 16303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:04.714463 16311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:04.715205 16311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:04.715337 16311 catalog_manager.cc:1259] Loaded cluster ID: f7fa62b6e77549fd81eacff50c421c2e
I20260928 23:20:04.715454 16311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:04.716953 16311 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:04.717109 16311 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Loaded TSK: 0
I20260928 23:20:04.726163 16311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Generated new TSK 1
I20260928 23:20:04.726251 16311 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:20:04.784660 16301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:04.784868 16301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:04.784898 16301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:04.786015 16301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:04.786054 16301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:04.786088 16301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:20:04.787503 16301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34001
--tserver_master_addrs=127.0.98.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:04.787750 16301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:04.787945 16301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:04.792667 16326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:04.792622 16327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:04.792791 16301 server_base.cc:1034] running on GCE node
W20260928 23:20:04.792634 16329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:04.793051 16301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:04.793283 16301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:04.794448 16301 hybrid_clock.cc:648] HybridClock initialized: now 1790637604794438 us; error 29 us; skew 500 ppm
Sep 28 23:20:04 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:04.797044 16301 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:20:04.797484 16301 webserver.cc:460] Webserver started at http://127.0.98.1:34001/ using document root <none> and password file <none>
I20260928 23:20:04.797672 16301 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:04.797732 16301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:04.799044 16301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:04.799754 16336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:04.799919 16301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:04.800004 16301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "ab568863eb584ef594447579a0aec55a"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "8f6b55a8ac2c4605fbd8d38d48b972c8"
server_key_iv: "d44d7b0ac84363808487467312d5acfd"
server_key_version: "encryptionkey@0"
I20260928 23:20:04.800338 16301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:04.807222 16301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:04.809142 16301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:04.809264 16301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:04.809442 16301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:04.809816 16343 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:20:04.811612 16301 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:20:04.811673 16301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 23:20:04.811717 16301 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:20:04.812309 16301 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:20:04.812340 16301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 23:20:04.812424 16343 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Bootstrap starting.
I20260928 23:20:04.818635 16301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45869
I20260928 23:20:04.818961 16301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:20:04.822906 16343 log.cc:824] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:04.823339 16450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45869 every 8 connection(s)
I20260928 23:20:04.826092 16343 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:20:04.826376 16343 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Bootstrap complete.
I20260928 23:20:04.826653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 16301
I20260928 23:20:04.826912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39533
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38957
--webserver_port=42267
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:20:04.827414 16343 ts_tablet_manager.cc:1397] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.008s
I20260928 23:20:04.829027 16343 raft_consensus.cc:348] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:20:04.829419 16343 raft_consensus.cc:729] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab568863eb584ef594447579a0aec55a, State: Initialized, Role: FOLLOWER
I20260928 23:20:04.829613 16343 consensus_queue.cc:260] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:20:04.829859 16343 ts_tablet_manager.cc:1428] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Time spent starting tablet: real 0.002s user 0.005s sys 0.001s
Sep 28 23:20:04 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:04.833633 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:04.819599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:34195)
0928 23:20:04.823255 (+ 3656us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:04.823304 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:04.823926 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:04.824426 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:04.824430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:04.824722 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:04.824901 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:04.824907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.826052 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.826053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:04.827156 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:04.827159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.828059 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.828061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:04.828220 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:04.828797 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:04.828808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:04.830366 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:04.832156 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:04.832164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:04.832165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:04.832385 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:04.832609 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:04.832610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:04.832611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:04.832721 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:04.833053 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:04.833059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:04.833185 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 23:20:04.833269 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:20:04.833439 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3608,"thread_start_us":91,"threads_started":1}
I20260928 23:20:04.833629 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:04.819641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34195 (local address 127.0.98.62:38957)
0928 23:20:04.823639 (+ 3998us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:04.823647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:04.823692 (+ 45us) server_negotiation.cc:407] Connection header received
0928 23:20:04.824188 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:04.824194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:04.824256 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:04.824385 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:04.825023 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.825934 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.827252 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.827854 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.827950 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:04.830481 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:04.830485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:04.830486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:04.830498 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:04.832080 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:04.832457 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:04.832461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:04.832461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:04.832551 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:04.832778 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:04.832779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:04.832779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:04.832991 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:04.833061 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:04.833327 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 23:20:04.833439 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3939,"thread_start_us":71,"threads_started":1}
I20260928 23:20:04.834558 16451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38957
I20260928 23:20:04.834656 16451 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:04.834815 16451 heartbeater.cc:511] Master 127.0.98.62:38957 requested a full tablet report, sending...
I20260928 23:20:04.835448 16263 ts_manager.cc:194] Registered new tserver with Master: ab568863eb584ef594447579a0aec55a (127.0.98.1:45869)
I20260928 23:20:04.836562 16263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34195
I20260928 23:20:04.837474 16451 heartbeater.cc:503] Master 127.0.98.62:38957 was elected leader, sending a full tablet report...
Sep 28 23:20:04 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:20:04.843775 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:04.836018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45869 (local address 127.0.0.1:44346)
0928 23:20:04.836375 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:04.836407 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:04.836463 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:04.836755 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:04.836759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:04.837240 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:04.837291 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:04.837293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.838412 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.838412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:04.838956 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:04.838957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.839381 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.839384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:04.839452 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:04.839876 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:04.839880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:04.841116 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:04.842600 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:04.842606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:04.842608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:04.842834 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:04.843091 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:04.843092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:04.843093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:04.843157 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:04.843399 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:04.843406 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:04.843457 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 23:20:04.843528 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 23:20:04.843684 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":80,"threads_started":1}
I20260928 23:20:04.843802 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:04.836241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.0.98.1:45869)
0928 23:20:04.836568 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:04.836572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:04.836580 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:20:04.836612 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:04.836614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:04.836640 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:04.836699 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:04.837377 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.838340 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.839031 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.839319 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.839375 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:04.841192 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:04.841199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:04.841199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:04.841206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:04.842524 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:04.842965 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:04.842968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:04.842969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:04.843032 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:04.843207 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:04.843208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:04.843208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:04.843337 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:04.843409 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:04.843588 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 23:20:04.843683 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":60,"threads_started":1}
I20260928 23:20:04.844455 16380 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bd46162a254d0ca631bd9e7588301d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44346
I20260928 23:20:04.844924 16460 tablet_replica.cc:330] stopping tablet replica
I20260928 23:20:04.845021 16460 raft_consensus.cc:2229] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:04.845090 16460 raft_consensus.cc:2258] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:04.845374 16460 ts_tablet_manager.cc:1905] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:04.847390 16460 ts_tablet_manager.cc:1918] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:04.847455 16460 log.cc:1196] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal/wals/14bd46162a254d0ca631bd9e7588301d
I20260928 23:20:04.847685 16460 ts_tablet_manager.cc:1939] T 14bd46162a254d0ca631bd9e7588301d P ab568863eb584ef594447579a0aec55a: Deleting consensus metadata
I20260928 23:20:04.848129 16249 catalog_manager.cc:4841] TS ab568863eb584ef594447579a0aec55a (127.0.98.1:45869): tablet 14bd46162a254d0ca631bd9e7588301d (table test-table [id=3d2e2d016a2b4e5fba37e1c369aeed4b]) successfully deleted
W20260928 23:20:04.910754 16456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:04.910943 16456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:04.910965 16456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:04.912086 16456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:04.912122 16456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:04.912147 16456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:20:04.913552 16456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42267
--tserver_master_addrs=127.0.98.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:04.913758 16456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:04.914011 16456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:04.918694 16468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:04.918812 16456 server_base.cc:1034] running on GCE node
W20260928 23:20:04.918684 16466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:04.918678 16465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:04.919075 16456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:04.919296 16456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:04.920434 16456 hybrid_clock.cc:648] HybridClock initialized: now 1790637604920421 us; error 29 us; skew 500 ppm
Sep 28 23:20:04 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:04.922796 16456 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:20:04.923156 16456 webserver.cc:460] Webserver started at http://127.0.98.2:42267/ using document root <none> and password file <none>
I20260928 23:20:04.923336 16456 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:04.923381 16456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:04.924567 16456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:04.925135 16475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:04.925266 16456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:04.925335 16456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "2321b542914c4e9484e45de0c2f5d781"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "11b1a29c4e3193b52b5bd82acb4b3562"
server_key_iv: "4628f6dbb200c755213653d25cd32844"
server_key_version: "encryptionkey@0"
I20260928 23:20:04.925637 16456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:04.932082 16456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:04.933984 16456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:04.934101 16456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:04.934278 16456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:04.934665 16482 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:20:04.936094 16456 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:20:04.936149 16456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 23:20:04.936182 16456 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:20:04.936795 16456 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:20:04.936829 16456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 23:20:04.936884 16482 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Bootstrap starting.
I20260928 23:20:04.942217 16456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39533
I20260928 23:20:04.942330 16589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39533 every 8 connection(s)
I20260928 23:20:04.942642 16456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:20:04.946635 16482 log.cc:824] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:04.949038 16482 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:20:04.949328 16482 tablet_bootstrap.cc:492] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Bootstrap complete.
Sep 28 23:20:04 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:04.950086 16482 ts_tablet_manager.cc:1397] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260928 23:20:04.950995 16482 raft_consensus.cc:348] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:20:04.951211 16482 raft_consensus.cc:729] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2321b542914c4e9484e45de0c2f5d781, State: Initialized, Role: FOLLOWER
I20260928 23:20:04.951329 16482 consensus_queue.cc:260] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } }
I20260928 23:20:04.951589 16482 ts_tablet_manager.cc:1428] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20260928 23:20:04.952715 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:04.943387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52911 (local address 127.0.98.62:38957)
0928 23:20:04.943451 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:04.943453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:04.943717 (+ 264us) server_negotiation.cc:407] Connection header received
0928 23:20:04.944361 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:04.944362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:04.944375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:04.944423 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:04.945364 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.945898 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.947312 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.947521 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.947638 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:04.950012 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:04.950014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:04.950014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:04.950022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:04.951343 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:04.951829 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:04.951830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:04.951831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:04.951885 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:04.952095 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:04.952096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:04.952096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:04.952214 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:04.952270 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:04.952514 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 23:20:04.952603 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:20:04.952829 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:04.943300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:52911)
0928 23:20:04.943526 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:04.943618 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:04.944239 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:04.944508 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:04.944513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:04.944935 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:04.945256 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:04.945265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.946015 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.946016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:04.947222 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:04.947224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.947663 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.947665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:04.947827 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:04.948346 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:04.948355 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:04.949908 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:04.951408 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:04.951415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:04.951417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:04.951755 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:04.951936 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:04.951936 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:04.951937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:04.952040 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:04.952268 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:04.952274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:04.952400 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 23:20:04.952486 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:20:04.952649 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
I20260928 23:20:04.952886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 16456
I20260928 23:20:04.953137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45153
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38957
--webserver_port=42111
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:20:04.953262 16590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38957
I20260928 23:20:04.953354 16590 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:04.953509 16590 heartbeater.cc:511] Master 127.0.98.62:38957 requested a full tablet report, sending...
I20260928 23:20:04.954025 16263 ts_manager.cc:194] Registered new tserver with Master: 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533)
I20260928 23:20:04.954586 16263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52911
I20260928 23:20:04.955293 16590 heartbeater.cc:503] Master 127.0.98.62:38957 was elected leader, sending a full tablet report...
Sep 28 23:20:04 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 23:20:04.961369 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:04.954483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.0.98.2:39533)
0928 23:20:04.954637 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:04.954641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:04.954654 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:20:04.954686 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:04.954688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:04.954702 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:04.954810 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:04.955087 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.956084 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.956731 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.957163 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.957249 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:04.958946 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:04.958952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:04.958953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:04.958958 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:04.960303 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:04.960688 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:04.960690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:04.960691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:04.960738 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:04.960916 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:04.960916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:04.960917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:04.961036 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:04.961091 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:04.961239 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 23:20:04.961308 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260928 23:20:04.961412 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:04.954312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39533 (local address 127.0.0.1:40562)
0928 23:20:04.954353 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:04.954363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:04.954392 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:04.954818 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:04.954818 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:04.954973 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:04.955014 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:04.955016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.956153 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.956154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:04.956659 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:04.956660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:04.957244 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:04.957246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:04.957296 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:04.957688 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:04.957692 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:04.958864 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:04.960382 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:04.960383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:04.960384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:04.960588 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:04.960795 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:04.960795 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:04.960796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:04.960862 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:04.961090 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:04.961091 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:04.961139 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 23:20:04.961216 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:20:04.961321 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:20:04.961791 16524 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bd46162a254d0ca631bd9e7588301d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:19:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40562
I20260928 23:20:04.962230 16597 tablet_replica.cc:330] stopping tablet replica
I20260928 23:20:04.962313 16597 raft_consensus.cc:2229] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:04.962376 16597 raft_consensus.cc:2258] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:04.962638 16597 ts_tablet_manager.cc:1905] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:04.964628 16597 ts_tablet_manager.cc:1918] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:04.964690 16597 log.cc:1196] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/14bd46162a254d0ca631bd9e7588301d
I20260928 23:20:04.964896 16597 ts_tablet_manager.cc:1939] T 14bd46162a254d0ca631bd9e7588301d P 2321b542914c4e9484e45de0c2f5d781: Deleting consensus metadata
I20260928 23:20:04.965296 16249 catalog_manager.cc:4841] TS 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533): tablet 14bd46162a254d0ca631bd9e7588301d (table test-table [id=3d2e2d016a2b4e5fba37e1c369aeed4b]) successfully deleted
W20260928 23:20:05.033367 16595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:05.033560 16595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:05.033581 16595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:05.034659 16595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:05.034693 16595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:05.034716 16595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:20:05.036103 16595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42111
--tserver_master_addrs=127.0.98.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:05.036278 16595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:05.036439 16595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:05.041201 16605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:05.041265 16602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:05.041225 16595 server_base.cc:1034] running on GCE node
W20260928 23:20:05.041208 16603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:05.041632 16595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:05.041880 16595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:05.043025 16595 hybrid_clock.cc:648] HybridClock initialized: now 1790637605043012 us; error 28 us; skew 500 ppm
Sep 28 23:20:05 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:05.045452 16595 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:20:05.045962 16595 webserver.cc:460] Webserver started at http://127.0.98.3:42111/ using document root <none> and password file <none>
I20260928 23:20:05.046177 16595 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:05.046240 16595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:05.047540 16595 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:05.048110 16612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:05.048256 16595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:05.048331 16595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "12d68848d62c49c19db5569ad31ccaf9"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "00f8033ac55a03e7a6e0375fb60e0339"
server_key_iv: "6c04d1ef622c20318984bd41e8a6779d"
server_key_version: "encryptionkey@0"
I20260928 23:20:05.048645 16595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:05.077661 16595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:05.079708 16595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:05.079845 16595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:05.080062 16595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:05.080360 16595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:05.080392 16595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:05.080420 16595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:05.080446 16595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:05.085929 16595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45153
I20260928 23:20:05.085976 16725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45153 every 8 connection(s)
I20260928 23:20:05.086289 16595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:20:05.088246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 16595
Sep 28 23:20:05 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:05.094452 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.088605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.0.98.62:38957)
0928 23:20:05.088703 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:05.088705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:05.088800 (+ 95us) server_negotiation.cc:407] Connection header received
0928 23:20:05.088843 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:05.088844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:05.088856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:05.088910 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:05.089438 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.089910 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.090639 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.090862 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.090925 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.092081 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:05.092082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:05.092083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:05.092089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:05.093470 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:05.093765 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:05.093766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:05.093767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:05.093817 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:05.093972 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:05.093973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:05.093974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:05.094078 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:05.094249 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:05.094287 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:20:05.094359 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":52,"threads_started":1}
I20260928 23:20:05.095364 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.086926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58945 (local address 127.0.98.62:38957)
0928 23:20:05.086981 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:05.086983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:05.087170 (+ 187us) server_negotiation.cc:407] Connection header received
0928 23:20:05.087677 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:05.087678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:05.087689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:05.087743 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:05.088784 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.089276 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.090216 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.090420 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.090474 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.092831 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:05.092833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:05.092833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:05.092839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:05.094036 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:05.094452 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:05.094453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:05.094453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:05.094519 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:05.094775 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:05.094776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:05.094776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:05.094890 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:05.094935 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:05.095229 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 23:20:05.095303 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:20:05.095592 16728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.086928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:58945)
0928 23:20:05.087056 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:05.087091 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:05.087586 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:05.087797 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:05.087800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:05.088466 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:05.088706 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:05.088713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.089360 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.089361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:05.090140 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:05.090142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.090486 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.090487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:05.090604 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.091095 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:05.091106 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:05.092744 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:05.094102 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:05.094109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:05.094111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:05.094383 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:05.094586 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:05.094587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:05.094587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:05.094714 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:05.094936 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:05.094942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:05.095097 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 23:20:05.095206 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:20:05.095354 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260928 23:20:05.095993 16726 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38957
I20260928 23:20:05.096073 16726 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:05.096210 16726 heartbeater.cc:511] Master 127.0.98.62:38957 requested a full tablet report, sending...
I20260928 23:20:05.096498 16263 ts_manager.cc:194] Registered new tserver with Master: 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153)
I20260928 23:20:05.096838 16263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58945
I20260928 23:20:05.105940 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:20:05.108444 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.106263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.0.98.62:38957)
0928 23:20:05.106310 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:05.106312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:05.106422 (+ 110us) server_negotiation.cc:407] Connection header received
0928 23:20:05.106474 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:05.106475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:05.106505 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:05.106567 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:20:05.106654 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.107090 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.107821 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.107992 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.108031 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.108129 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:20:05.108169 (+ 40us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 23:20:05.108381 (+ 212us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33700: token expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 23:20:05.108495 16454 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33700: token expired
W20260928 23:20:05.108865 16196 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:38957)
I20260928 23:20:05.115034 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.109073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.98.62:38957)
0928 23:20:05.109116 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:05.109118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:05.109126 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:20:05.109206 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:05.109207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:05.109214 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:05.109252 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:05.109793 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.110224 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.111169 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.111457 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.111519 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.112615 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:05.112616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:05.112617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:05.112623 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:05.113897 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:05.114275 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:05.114276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:05.114277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:05.114328 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:05.114577 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:05.114578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:05.114578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:05.114697 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:05.114794 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:05.114889 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 23:20:05.114955 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:20:05.116035 16263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:20:05.117590 16263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:20:05.119375 16380 tablet_service.cc:1467] Processing CreateTablet for tablet 70f0e2c540324569a50745cdbca175e1 (DEFAULT_TABLE table=test-table [id=b6667d68f8f84472870bbc5da3a7abb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:05.119408 16524 tablet_service.cc:1467] Processing CreateTablet for tablet 70f0e2c540324569a50745cdbca175e1 (DEFAULT_TABLE table=test-table [id=b6667d68f8f84472870bbc5da3a7abb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:05.119581 16380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f0e2c540324569a50745cdbca175e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:05.119601 16524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f0e2c540324569a50745cdbca175e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:05.120927 16482 tablet_bootstrap.cc:492] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: Bootstrap starting.
I20260928 23:20:05.121033 16343 tablet_bootstrap.cc:492] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Bootstrap starting.
I20260928 23:20:05.121208 16482 tablet_bootstrap.cc:654] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:05.121320 16343 tablet_bootstrap.cc:654] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:05.121925 16482 tablet_bootstrap.cc:492] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: No bootstrap required, opened a new log
I20260928 23:20:05.121968 16482 ts_tablet_manager.cc:1397] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:05.122097 16482 raft_consensus.cc:348] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:05.122151 16482 raft_consensus.cc:374] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:05.122222 16482 raft_consensus.cc:729] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2321b542914c4e9484e45de0c2f5d781, State: Initialized, Role: FOLLOWER
I20260928 23:20:05.122267 16482 consensus_queue.cc:260] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:05.122354 16482 ts_tablet_manager.cc:1428] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:05.122671 16343 tablet_bootstrap.cc:492] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: No bootstrap required, opened a new log
I20260928 23:20:05.122713 16343 ts_tablet_manager.cc:1397] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 23:20:05.122845 16343 raft_consensus.cc:348] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:05.122913 16343 raft_consensus.cc:374] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:05.122960 16343 raft_consensus.cc:729] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab568863eb584ef594447579a0aec55a, State: Initialized, Role: FOLLOWER
I20260928 23:20:05.123006 16343 consensus_queue.cc:260] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:05.123088 16343 ts_tablet_manager.cc:1428] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:05.123189 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.119148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45153 (local address 127.0.0.1:57948)
0928 23:20:05.119201 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:05.119210 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:05.119245 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:05.119736 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:05.119737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:05.119738 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:05.119785 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:05.119786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.121031 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.121032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:05.122113 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:05.122115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.122976 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.122977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:05.123037 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.123060 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 23:20:05.123089 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 23:20:05.123127 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 23:20:05.123497 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.119322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.0.98.3:45153)
0928 23:20:05.119509 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:05.119514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:05.119525 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:20:05.119602 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:05.119605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:05.119621 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:05.119855 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:05.119938 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.120934 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.122192 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.122893 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.123281 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.123353 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:05.123378 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 23:20:05.123415 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260928 23:20:05.124107 16660 tablet_service.cc:1467] Processing CreateTablet for tablet 70f0e2c540324569a50745cdbca175e1 (DEFAULT_TABLE table=test-table [id=b6667d68f8f84472870bbc5da3a7abb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:05.124424 16660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f0e2c540324569a50745cdbca175e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:05.126933 16735 tablet_bootstrap.cc:492] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: Bootstrap starting.
I20260928 23:20:05.127538 16735 tablet_bootstrap.cc:654] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:05.128046 16735 log.cc:824] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:05.128912 16735 tablet_bootstrap.cc:492] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: No bootstrap required, opened a new log
I20260928 23:20:05.128973 16735 ts_tablet_manager.cc:1397] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:20:05.130286 16735 raft_consensus.cc:348] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:05.130419 16735 raft_consensus.cc:374] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:05.130440 16735 raft_consensus.cc:729] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d68848d62c49c19db5569ad31ccaf9, State: Initialized, Role: FOLLOWER
I20260928 23:20:05.130553 16735 consensus_queue.cc:260] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:05.130836 16735 ts_tablet_manager.cc:1428] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:20:05.130959 16726 heartbeater.cc:503] Master 127.0.98.62:38957 was elected leader, sending a full tablet report...
W20260928 23:20:05.197165 16591 tablet.cc:2369] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:20:05.323880 16452 tablet.cc:2369] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:20:05.336800 16727 tablet.cc:2369] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:20:05.369942 16457 raft_consensus.cc:482] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:05.370078 16457 raft_consensus.cc:504] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:05.370437 16457 leader_election.cc:288] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153), 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533)
I20260928 23:20:05.374341 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.370690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54929 (local address 127.0.98.3:45153)
0928 23:20:05.370733 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:05.370736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:05.370843 (+ 107us) server_negotiation.cc:407] Connection header received
0928 23:20:05.370892 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:05.370893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:05.370905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:05.370970 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:05.371222 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.371992 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.373433 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.373989 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.374061 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.374073 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:05.374232 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 23:20:05.374282 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:20:05.374342 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.370750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34213 (local address 127.0.98.2:39533)
0928 23:20:05.370792 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:05.370794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:05.370804 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:20:05.370893 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:05.370894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:05.370919 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:05.370986 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:05.371222 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.372008 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.373433 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.374059 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.374117 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.374173 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:05.374232 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 23:20:05.374270 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:20:05.374373 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.370614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45153 (local address 127.0.98.1:54929)
0928 23:20:05.370772 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:05.370780 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:05.370834 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:05.371060 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:05.371061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:05.371064 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:05.371146 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:05.371149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.372074 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.372075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:05.373346 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:05.373349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.374069 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.374070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:05.374130 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.374160 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:20:05.374219 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:20:05.374266 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":30,"thread_start_us":64,"threads_started":1}
I20260928 23:20:05.374377 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.370613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39533 (local address 127.0.98.1:34213)
0928 23:20:05.370725 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:05.370736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:05.370816 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:05.371060 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:05.371062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:05.371066 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:05.371146 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:05.371149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.372074 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.372075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:05.373346 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:05.373349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.374124 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.374124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:05.374169 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.374171 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:20:05.374230 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:20:05.374266 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":88,"thread_start_us":63,"threads_started":1}
I20260928 23:20:05.374769 16544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f0e2c540324569a50745cdbca175e1" candidate_uuid: "ab568863eb584ef594447579a0aec55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2321b542914c4e9484e45de0c2f5d781" is_pre_election: true
I20260928 23:20:05.374923 16544 raft_consensus.cc:2454] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab568863eb584ef594447579a0aec55a in term 0.
I20260928 23:20:05.374893 16680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f0e2c540324569a50745cdbca175e1" candidate_uuid: "ab568863eb584ef594447579a0aec55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d68848d62c49c19db5569ad31ccaf9" is_pre_election: true
I20260928 23:20:05.375030 16680 raft_consensus.cc:2454] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab568863eb584ef594447579a0aec55a in term 0.
I20260928 23:20:05.375136 16339 leader_election.cc:302] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2321b542914c4e9484e45de0c2f5d781, ab568863eb584ef594447579a0aec55a; no voters:
I20260928 23:20:05.375254 16457 raft_consensus.cc:2781] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:20:05.375306 16457 raft_consensus.cc:482] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:20:05.375334 16457 raft_consensus.cc:3037] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:05.376017 16457 raft_consensus.cc:504] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:05.376138 16457 leader_election.cc:288] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 1 election: Requested vote from peers 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153), 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533)
I20260928 23:20:05.376284 16680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f0e2c540324569a50745cdbca175e1" candidate_uuid: "ab568863eb584ef594447579a0aec55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d68848d62c49c19db5569ad31ccaf9"
I20260928 23:20:05.376287 16544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f0e2c540324569a50745cdbca175e1" candidate_uuid: "ab568863eb584ef594447579a0aec55a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2321b542914c4e9484e45de0c2f5d781"
I20260928 23:20:05.376355 16680 raft_consensus.cc:3037] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:05.376355 16544 raft_consensus.cc:3037] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:05.376974 16544 raft_consensus.cc:2454] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab568863eb584ef594447579a0aec55a in term 1.
I20260928 23:20:05.377003 16680 raft_consensus.cc:2454] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab568863eb584ef594447579a0aec55a in term 1.
I20260928 23:20:05.377151 16338 leader_election.cc:302] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d68848d62c49c19db5569ad31ccaf9, ab568863eb584ef594447579a0aec55a; no voters:
I20260928 23:20:05.377251 16457 raft_consensus.cc:2781] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:05.377341 16457 raft_consensus.cc:686] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 LEADER]: Becoming Leader. State: Replica: ab568863eb584ef594447579a0aec55a, State: Running, Role: LEADER
I20260928 23:20:05.377398 16457 consensus_queue.cc:237] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:05.378144 16263 catalog_manager.cc:5495] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a reported cstate change: term changed from 0 to 1, leader changed from <none> to ab568863eb584ef594447579a0aec55a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ab568863eb584ef594447579a0aec55a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } health_report { overall_health: UNKNOWN } } }
I20260928 23:20:05.403820 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.401601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.0.98.1:45869)
0928 23:20:05.401742 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:05.401744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:05.401751 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:20:05.401792 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:05.401793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:05.401812 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:05.401882 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:20:05.401976 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.402446 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.403108 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:05.403314 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:05.403367 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:05.403486 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:20:05.403617 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:20:05.403684 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:05.403736 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:20:05.403753 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260928 23:20:05.405206 16680 raft_consensus.cc:1261] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9 [term 1 FOLLOWER]: Refusing update from remote peer ab568863eb584ef594447579a0aec55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:05.405470 16544 raft_consensus.cc:1261] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Refusing update from remote peer ab568863eb584ef594447579a0aec55a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:05.405514 16740 consensus_queue.cc:1035] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:05.405745 16457 consensus_queue.cc:1035] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:05.407914 16743 mvcc.cc:205] Tried to move back new op lower bound from 7334451631737532416 to 7334451631626346496. Current Snapshot: MvccSnapshot[applied={T|T < 7334451631737532416}]
I20260928 23:20:05.408613 16744 mvcc.cc:205] Tried to move back new op lower bound from 7334451631737532416 to 7334451631626346496. Current Snapshot: MvccSnapshot[applied={T|T < 7334451631737532416 or (T in {7334451631737532416})}]
I20260928 23:20:05.409772 16263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:05.409864 16263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:05.410516 16263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Sending DeleteTablet for 3 replicas of tablet 70f0e2c540324569a50745cdbca175e1
I20260928 23:20:05.410792 16660 tablet_service.cc:1514] Processing DeleteTablet for tablet 70f0e2c540324569a50745cdbca175e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57948
I20260928 23:20:05.410807 16380 tablet_service.cc:1514] Processing DeleteTablet for tablet 70f0e2c540324569a50745cdbca175e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44346
I20260928 23:20:05.410871 16524 tablet_service.cc:1514] Processing DeleteTablet for tablet 70f0e2c540324569a50745cdbca175e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40562
I20260928 23:20:05.410944 16597 tablet_replica.cc:330] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: stopping tablet replica
I20260928 23:20:05.410982 16597 raft_consensus.cc:2229] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:05.411032 16597 raft_consensus.cc:2258] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:05.411082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 16301
I20260928 23:20:05.413285 16597 ts_tablet_manager.cc:1905] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:05.414501 16754 tablet_replica.cc:330] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: stopping tablet replica
I20260928 23:20:05.414602 16754 raft_consensus.cc:2229] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:05.414702 16754 raft_consensus.cc:2258] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:05.414983 16597 ts_tablet_manager.cc:1918] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:05.415036 16597 log.cc:1196] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/70f0e2c540324569a50745cdbca175e1
I20260928 23:20:05.415182 16754 ts_tablet_manager.cc:1905] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:05.415216 16597 ts_tablet_manager.cc:1939] T 70f0e2c540324569a50745cdbca175e1 P 2321b542914c4e9484e45de0c2f5d781: Deleting consensus metadata
I20260928 23:20:05.415489 16249 catalog_manager.cc:4841] TS 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533): tablet 70f0e2c540324569a50745cdbca175e1 (table test-table [id=b6667d68f8f84472870bbc5da3a7abb1]) successfully deleted
I20260928 23:20:05.416708 16754 ts_tablet_manager.cc:1918] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:05.416764 16754 log.cc:1196] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/70f0e2c540324569a50745cdbca175e1
I20260928 23:20:05.416990 16754 ts_tablet_manager.cc:1939] T 70f0e2c540324569a50745cdbca175e1 P 12d68848d62c49c19db5569ad31ccaf9: Deleting consensus metadata
I20260928 23:20:05.417330 16248 catalog_manager.cc:4841] TS 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153): tablet 70f0e2c540324569a50745cdbca175e1 (table test-table [id=b6667d68f8f84472870bbc5da3a7abb1]) successfully deleted
W20260928 23:20:05.418234 16249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45869 (error 108)
I20260928 23:20:05.418336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 16456
I20260928 23:20:05.422505 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:05.422370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45869 (local address 127.0.0.1:44366)
0928 23:20:05.422431 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:05.422475 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 23:20:05.422536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 16595
W20260928 23:20:05.422633 16249 catalog_manager.cc:4584] TS ab568863eb584ef594447579a0aec55a (127.0.98.1:45869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70f0e2c540324569a50745cdbca175e1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45869: connect: Connection refused (error 111)
I20260928 23:20:05.426393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 16230
I20260928 23:20:25.430768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38957
--webserver_interface=127.0.98.62
--webserver_port=46007
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:25.508754 16756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:25.508953 16756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:25.508975 16756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:25.510281 16756 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:20:25.510334 16756 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:20:25.510346 16756 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:20:25.510358 16756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:25.510370 16756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:25.510381 16756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:25.510421 16756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:25.510437 16756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:25.512091 16756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:25.512331 16756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:25.512552 16756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:25.517414 16761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:25.517390 16764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:25.517452 16756 server_base.cc:1034] running on GCE node
W20260928 23:20:25.517411 16762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:25.517839 16756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:25.518121 16756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:25.519284 16756 hybrid_clock.cc:648] HybridClock initialized: now 1790637625519267 us; error 28 us; skew 500 ppm
Sep 28 23:20:25 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:25.521704 16756 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:20:25.522100 16756 webserver.cc:460] Webserver started at http://127.0.98.62:46007/ using document root <none> and password file <none>
I20260928 23:20:25.522301 16756 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:25.522348 16756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:25.523656 16756 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:25.524278 16771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:25.524417 16756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:25.524494 16756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "93aadaee7e254ccb8f1342ac914b2a50"
format_stamp: "Formatted at 2026-09-28 23:19:43 on dist-test-slave-djkr"
server_key: "b4931dfd9b34918fe51c8a11f08ee765"
server_key_iv: "4735543c2a4b46ae03dc07d47ed1278a"
server_key_version: "encryptionkey@0"
I20260928 23:20:25.524827 16756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:25.545339 16756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:25.547351 16756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:25.547487 16756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:25.551126 16756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38957
I20260928 23:20:25.551165 16823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38957 every 8 connection(s)
I20260928 23:20:25.551502 16756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:25.553012 16824 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:20:25.553822 16824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Bootstrap starting.
I20260928 23:20:25.555451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 16756
I20260928 23:20:25.555778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45869
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38957
--webserver_port=34001
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:20:25.562225 16824 log.cc:824] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:25.564370 16824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:20:25.564630 16824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Bootstrap complete.
I20260928 23:20:25.566610 16824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:20:25.566835 16824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93aadaee7e254ccb8f1342ac914b2a50, State: Initialized, Role: FOLLOWER
I20260928 23:20:25.566979 16824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:20:25.567056 16824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:25.567099 16824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:25.567147 16824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 2 FOLLOWER]: Advancing to term 3
I20260928 23:20:25.568166 16824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:20:25.568259 16824 leader_election.cc:302] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93aadaee7e254ccb8f1342ac914b2a50; no voters:
I20260928 23:20:25.568416 16824 leader_election.cc:288] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 23:20:25.568463 16828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 23:20:25.568635 16828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [term 3 LEADER]: Becoming Leader. State: Replica: 93aadaee7e254ccb8f1342ac914b2a50, State: Running, Role: LEADER
I20260928 23:20:25.568672 16824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:25.568794 16828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } }
I20260928 23:20:25.569095 16830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93aadaee7e254ccb8f1342ac914b2a50. Latest consensus state: current_term: 3 leader_uuid: "93aadaee7e254ccb8f1342ac914b2a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } } }
I20260928 23:20:25.569113 16829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "93aadaee7e254ccb8f1342ac914b2a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93aadaee7e254ccb8f1342ac914b2a50" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38957 } } }
I20260928 23:20:25.569253 16829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:25.569209 16830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:25.569738 16836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:25.570502 16836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:25.570683 16836 catalog_manager.cc:1259] Loaded cluster ID: f7fa62b6e77549fd81eacff50c421c2e
I20260928 23:20:25.570804 16836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:25.572038 16836 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:25.572227 16836 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Loaded TSK: 0
I20260928 23:20:25.572279 16836 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93aadaee7e254ccb8f1342ac914b2a50: Loaded TSK: 1
F20260928 23:20:25.572523 16836 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ffb1e13a9ad google::LogMessage::Fail() at ??:0
@ 0x7ffb1e13d88a google::LogMessage::SendToLog() at ??:0
@ 0x7ffb1e13a4de google::LogMessage::Flush() at ??:0
@ 0x7ffb1e13ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffb208a7104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ffb208a4d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ffb260941cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ffb2609448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ffb26094594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ffb2605ac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ffb26064ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ffb200080d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ffb1fffd954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffb20e616db start_thread at ??:0
@ 0x7ffb1cc3861f clone at ??:0
W20260928 23:20:25.666263 16826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:25.666528 16826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:25.666563 16826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:25.668040 16826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:25.668087 16826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:25.668120 16826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:20:25.669524 16826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34001
--tserver_master_addrs=127.0.98.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:25.669757 16826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:25.669967 16826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:25.674664 16865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:25.674650 16866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:25.674640 16868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:25.674937 16826 server_base.cc:1034] running on GCE node
I20260928 23:20:25.675144 16826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:25.675405 16826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:25.676578 16826 hybrid_clock.cc:648] HybridClock initialized: now 1790637625676548 us; error 44 us; skew 500 ppm
Sep 28 23:20:25 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:25.679430 16826 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:20:25.679991 16826 webserver.cc:460] Webserver started at http://127.0.98.1:34001/ using document root <none> and password file <none>
I20260928 23:20:25.680244 16826 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:25.680317 16826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:25.681984 16826 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:20:25.682704 16875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:25.682875 16826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:20:25.682971 16826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "ab568863eb584ef594447579a0aec55a"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "8f6b55a8ac2c4605fbd8d38d48b972c8"
server_key_iv: "d44d7b0ac84363808487467312d5acfd"
server_key_version: "encryptionkey@0"
I20260928 23:20:25.683404 16826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:25.697381 16826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:25.699563 16826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:25.699714 16826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:25.699977 16826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:25.700526 16882 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:20:25.701937 16826 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:20:25.701995 16826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 23:20:25.702028 16826 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:20:25.702869 16826 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:20:25.702913 16826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 23:20:25.702975 16882 tablet_bootstrap.cc:492] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Bootstrap starting.
I20260928 23:20:25.709815 16826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45869
I20260928 23:20:25.709909 16989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45869 every 8 connection(s)
I20260928 23:20:25.710314 16826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:20:25.710795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 16826
I20260928 23:20:25.711092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39533
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38957
--webserver_port=42267
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:20:25.713024 16882 log.cc:824] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:25.715603 16882 tablet_bootstrap.cc:492] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:20:25.715968 16882 tablet_bootstrap.cc:492] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Bootstrap complete.
I20260928 23:20:25.717087 16882 ts_tablet_manager.cc:1397] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260928 23:20:25.718547 16882 raft_consensus.cc:348] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:25.718966 16882 raft_consensus.cc:729] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab568863eb584ef594447579a0aec55a, State: Initialized, Role: FOLLOWER
I20260928 23:20:25.719172 16882 consensus_queue.cc:260] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:25.719525 16882 ts_tablet_manager.cc:1428] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260928 23:20:25.778980 16992 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:20:25.712027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:32851)
0928 23:20:25.712215 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:25.712263 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:25.712868 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:25.778764 (+ 65896us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
W20260928 23:20:25.779237 16990 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38957 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38957: Client connection negotiation failed: client connection to 127.0.98.62:38957: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:20:25.779592 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:25.779447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:53747)
0928 23:20:25.779499 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:25.779540 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 23:20:25.779979 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:25.779886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:34827)
0928 23:20:25.779927 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:25.779955 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 23:20:25.780083 16990 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:20:25.793582 16993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:25.793764 16993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:25.793787 16993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:25.794865 16993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:25.794898 16993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:25.794921 16993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:20:25.796382 16993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42267
--tserver_master_addrs=127.0.98.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:25.796555 16993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:25.796726 16993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:25.801343 17003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:25.801359 17001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:25.801343 17000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:25.801501 16993 server_base.cc:1034] running on GCE node
I20260928 23:20:25.801764 16993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:25.801994 16993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:25.803128 16993 hybrid_clock.cc:648] HybridClock initialized: now 1790637625803116 us; error 28 us; skew 500 ppm
Sep 28 23:20:25 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:25.805442 16993 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:20:25.805787 16993 webserver.cc:460] Webserver started at http://127.0.98.2:42267/ using document root <none> and password file <none>
I20260928 23:20:25.805961 16993 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:25.806000 16993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:25.807192 16993 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:20:25.807714 17010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:25.807857 16993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:25.807928 16993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "2321b542914c4e9484e45de0c2f5d781"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "11b1a29c4e3193b52b5bd82acb4b3562"
server_key_iv: "4628f6dbb200c755213653d25cd32844"
server_key_version: "encryptionkey@0"
I20260928 23:20:25.808239 16993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:25.825948 16993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:25.827678 16993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:25.827804 16993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:25.828017 16993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:25.828300 16993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:25.828332 16993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:25.828353 16993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:25.828377 16993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:25.833666 16993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39533
I20260928 23:20:25.833741 17123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39533 every 8 connection(s)
I20260928 23:20:25.834015 16993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:20:25.834947 17126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:25.834597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:51169)
0928 23:20:25.834742 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:25.834812 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
W20260928 23:20:25.835314 17124 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38957 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38957: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
I20260928 23:20:25.835536 17126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:25.835460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:55133)
0928 23:20:25.835495 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:25.835520 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 23:20:25.835820 17126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:25.835735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:50973)
0928 23:20:25.835768 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:25.835806 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 23:20:25.835918 17124 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:20:25.836864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 16993
I20260928 23:20:25.837078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45153
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38957
--webserver_port=42111
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:25.914535 17127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:25.914734 17127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:25.914755 17127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:25.915853 17127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:25.915891 17127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:25.915916 17127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:20:25.917271 17127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42111
--tserver_master_addrs=127.0.98.62:38957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:25.917492 17127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:25.917675 17127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:25.922192 17132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:25.922217 17133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:25.922250 17127 server_base.cc:1034] running on GCE node
W20260928 23:20:25.922206 17135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:25.922583 17127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:25.922808 17127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:25.923943 17127 hybrid_clock.cc:648] HybridClock initialized: now 1790637625923928 us; error 33 us; skew 500 ppm
Sep 28 23:20:25 dist-test-slave-djkr krb5kdc[15687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:25.926110 17127 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:20:25.926484 17127 webserver.cc:460] Webserver started at http://127.0.98.3:42111/ using document root <none> and password file <none>
I20260928 23:20:25.926676 17127 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:25.926723 17127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:25.927971 17127 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:25.928460 17142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:25.928597 17127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:25.928670 17127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "12d68848d62c49c19db5569ad31ccaf9"
format_stamp: "Formatted at 2026-09-28 23:19:44 on dist-test-slave-djkr"
server_key: "00f8033ac55a03e7a6e0375fb60e0339"
server_key_iv: "6c04d1ef622c20318984bd41e8a6779d"
server_key_version: "encryptionkey@0"
I20260928 23:20:25.928951 17127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:25.936586 17127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:25.938211 17127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:25.938346 17127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:25.938534 17127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:25.938807 17127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:25.938838 17127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:25.938864 17127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:25.938890 17127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:25.944408 17127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45153
I20260928 23:20:25.944469 17255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45153 every 8 connection(s)
I20260928 23:20:25.944783 17127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:20:25.945742 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:25.945358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:48603)
0928 23:20:25.945503 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:25.945578 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
W20260928 23:20:25.946086 17256 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38957 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38957: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
I20260928 23:20:25.946333 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:25.946248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:50891)
0928 23:20:25.946293 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:25.946317 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 23:20:25.946557 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:25.946491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:45635)
0928 23:20:25.946520 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:25.946541 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 23:20:25.946645 17256 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:20:25.951251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 17127
I20260928 23:20:26.780632 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:26.780372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:59319)
0928 23:20:26.780517 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:26.780567 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
I20260928 23:20:26.836400 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:26.836186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:55195)
0928 23:20:26.836319 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:26.836351 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260928 23:20:26.947139 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:26.946944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:48601)
0928 23:20:26.947061 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:26.947093 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260928 23:20:27.232306 17263 raft_consensus.cc:482] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:27.232451 17263 raft_consensus.cc:504] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:27.232827 17263 leader_election.cc:288] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153), 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533)
Sep 28 23:20:27 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 28 23:20:27 dist-test-slave-djkr krb5kdc[15687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 23:20:27.241971 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:27.233057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37847 (local address 127.0.98.2:39533)
0928 23:20:27.233220 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:27.233235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:27.233271 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:20:27.233695 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:27.233696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:27.233735 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:27.233853 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:27.234555 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:27.235395 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:27.236346 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:27.236723 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:27.236796 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:27.239253 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:27.239257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:27.239257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:27.239269 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:27.240763 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:27.241062 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:27.241065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:27.241065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:27.241142 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:27.241316 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:27.241317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:27.241317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:27.241522 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:27.241586 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:27.241749 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 23:20:27.241862 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260928 23:20:27.241978 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:27.233093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42281 (local address 127.0.98.3:45153)
0928 23:20:27.233237 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:27.233247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:27.233273 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:20:27.233693 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:27.233695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:27.233735 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:27.233864 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:27.234557 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:27.235382 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:27.236348 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:27.236766 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:27.236872 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:27.238873 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:27.238876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:27.238877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:27.238886 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:27.240536 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:27.240904 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:27.240907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:27.240907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:27.240989 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:27.241196 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:27.241197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:27.241197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:27.241412 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:27.241480 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:27.241749 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 23:20:27.241868 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260928 23:20:27.241971 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:27.232947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45153 (local address 127.0.98.1:42281)
0928 23:20:27.233007 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:27.233018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:27.233076 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:27.233911 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:27.233914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:27.234270 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:27.234446 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:27.234451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:27.235470 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:27.235471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:27.236251 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:27.236253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:27.236846 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:27.236847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:27.236909 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:27.237411 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:27.237420 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:27.238772 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:27.240618 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:27.240625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:27.240627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:27.240832 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:27.241045 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:27.241046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:27.241046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:27.241141 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:27.241486 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:27.241492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:27.241610 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 23:20:27.241694 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:20:27.241879 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":2}
I20260928 23:20:27.242067 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:27.232967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39533 (local address 127.0.98.1:37847)
0928 23:20:27.233083 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:27.233093 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:27.233144 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:27.233912 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:27.233914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:27.234271 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:27.234446 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:27.234451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:27.235477 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:27.235477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:27.236252 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:27.236254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:27.236804 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:27.236805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:27.236910 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:27.237454 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:27.237458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:27.239141 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:27.240842 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:27.240843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:27.240843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:27.240997 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:27.241200 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:27.241201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:27.241201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:27.241267 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:27.241581 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:27.241582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:27.241624 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:20:27.241696 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:20:27.241950 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":181,"thread_start_us":55,"threads_started":1}
W20260928 23:20:27.242823 16877 leader_election.cc:341] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153): Not found: Tablet not found: 70f0e2c540324569a50745cdbca175e1
W20260928 23:20:27.243183 16878 leader_election.cc:341] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533): Not found: Tablet not found: 70f0e2c540324569a50745cdbca175e1
I20260928 23:20:27.243223 16878 leader_election.cc:302] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab568863eb584ef594447579a0aec55a; no voters: 12d68848d62c49c19db5569ad31ccaf9, 2321b542914c4e9484e45de0c2f5d781
I20260928 23:20:27.243341 17263 raft_consensus.cc:2726] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:20:27.781227 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:27.781031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:46461)
0928 23:20:27.781145 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:27.781180 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260928 23:20:27.837006 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:27.836814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:60949)
0928 23:20:27.836936 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:27.836972 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260928 23:20:27.947664 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:27.947478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:32921)
0928 23:20:27.947591 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:27.947626 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260928 23:20:28.781827 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:28.781610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:60499)
0928 23:20:28.781746 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:28.781781 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260928 23:20:28.815128 17271 raft_consensus.cc:482] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:28.815220 17271 raft_consensus.cc:504] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:28.815356 17271 leader_election.cc:288] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153), 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533)
W20260928 23:20:28.815768 16878 leader_election.cc:341] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533): Not found: Tablet not found: 70f0e2c540324569a50745cdbca175e1
W20260928 23:20:28.815853 16877 leader_election.cc:341] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153): Not found: Tablet not found: 70f0e2c540324569a50745cdbca175e1
I20260928 23:20:28.815898 16877 leader_election.cc:302] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab568863eb584ef594447579a0aec55a; no voters: 12d68848d62c49c19db5569ad31ccaf9, 2321b542914c4e9484e45de0c2f5d781
I20260928 23:20:28.815975 17271 raft_consensus.cc:2726] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:20:28.837633 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:28.837425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:53063)
0928 23:20:28.837555 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:28.837590 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260928 23:20:28.948257 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:28.948065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:51245)
0928 23:20:28.948175 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:28.948216 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260928 23:20:29.782433 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:29.782236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:34529)
0928 23:20:29.782354 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:29.782386 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260928 23:20:29.838186 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:29.838002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:37873)
0928 23:20:29.838115 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:29.838148 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260928 23:20:29.948874 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:29.948683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:51489)
0928 23:20:29.948800 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:29.948835 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260928 23:20:30.783152 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:30.782907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:54711)
0928 23:20:30.783048 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:30.783106 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260928 23:20:30.838779 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:30.838583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:51863)
0928 23:20:30.838698 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:30.838740 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260928 23:20:30.949446 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:30.949244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:33867)
0928 23:20:30.949373 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:30.949405 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260928 23:20:31.553610 17280 raft_consensus.cc:482] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:31.553701 17280 raft_consensus.cc:504] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:31.553952 17280 leader_election.cc:288] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153), 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533)
W20260928 23:20:31.554193 16878 leader_election.cc:341] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533): Not found: Tablet not found: 70f0e2c540324569a50745cdbca175e1
W20260928 23:20:31.554266 16877 leader_election.cc:341] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153): Not found: Tablet not found: 70f0e2c540324569a50745cdbca175e1
I20260928 23:20:31.554335 16877 leader_election.cc:302] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab568863eb584ef594447579a0aec55a; no voters: 12d68848d62c49c19db5569ad31ccaf9, 2321b542914c4e9484e45de0c2f5d781
I20260928 23:20:31.554411 17280 raft_consensus.cc:2726] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:20:31.783802 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:31.783618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:38251)
0928 23:20:31.783727 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:31.783757 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260928 23:20:31.839386 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:31.839188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:45543)
0928 23:20:31.839309 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:31.839345 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260928 23:20:31.950023 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:31.949819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:56477)
0928 23:20:31.949947 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:31.949983 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260928 23:20:32.784415 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:32.784210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:42473)
0928 23:20:32.784341 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:32.784369 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260928 23:20:32.839998 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:32.839793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:35007)
0928 23:20:32.839915 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:32.839955 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260928 23:20:32.950639 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:32.950434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:33301)
0928 23:20:32.950560 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:32.950596 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260928 23:20:33.785102 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:33.784887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:59181)
0928 23:20:33.785018 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:33.785053 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260928 23:20:33.840549 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:33.840355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:38549)
0928 23:20:33.840478 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:33.840512 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260928 23:20:33.951254 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:33.951033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:33623)
0928 23:20:33.951160 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:33.951197 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260928 23:20:34.230650 17290 raft_consensus.cc:482] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:34.230756 17290 raft_consensus.cc:504] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d68848d62c49c19db5569ad31ccaf9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45153 } } peers { permanent_uuid: "ab568863eb584ef594447579a0aec55a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "2321b542914c4e9484e45de0c2f5d781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39533 } }
I20260928 23:20:34.230959 17290 leader_election.cc:288] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153), 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533)
W20260928 23:20:34.231343 16878 leader_election.cc:341] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2321b542914c4e9484e45de0c2f5d781 (127.0.98.2:39533): Not found: Tablet not found: 70f0e2c540324569a50745cdbca175e1
W20260928 23:20:34.231415 16877 leader_election.cc:341] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12d68848d62c49c19db5569ad31ccaf9 (127.0.98.3:45153): Not found: Tablet not found: 70f0e2c540324569a50745cdbca175e1
I20260928 23:20:34.231453 16877 leader_election.cc:302] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab568863eb584ef594447579a0aec55a; no voters: 12d68848d62c49c19db5569ad31ccaf9, 2321b542914c4e9484e45de0c2f5d781
I20260928 23:20:34.231521 17290 raft_consensus.cc:2726] T 70f0e2c540324569a50745cdbca175e1 P ab568863eb584ef594447579a0aec55a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:20:34.785772 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:34.785527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:51513)
0928 23:20:34.785681 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:34.785726 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":62,"threads_started":1}
I20260928 23:20:34.841152 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:34.840946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:57439)
0928 23:20:34.841063 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:34.841099 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260928 23:20:34.952003 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:34.951767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:43151)
0928 23:20:34.951911 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:34.951956 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260928 23:20:35.786450 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:35.786191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:56611)
0928 23:20:35.786363 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:35.786403 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260928 23:20:35.841791 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:35.841566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:54395)
0928 23:20:35.841703 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:35.841746 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260928 23:20:35.952636 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:35.952447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:33131)
0928 23:20:35.952559 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:35.952591 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260928 23:20:36.787225 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:36.787000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:39075)
0928 23:20:36.787134 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:36.787179 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260928 23:20:36.842634 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:36.842339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:41723)
0928 23:20:36.842524 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:36.842577 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260928 23:20:36.953331 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:36.953098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:32805)
0928 23:20:36.953235 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:36.953276 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260928 23:20:37.787931 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:37.787690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:56531)
0928 23:20:37.787850 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:37.787886 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
I20260928 23:20:37.843268 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:37.843073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:37789)
0928 23:20:37.843189 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:37.843225 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260928 23:20:37.954006 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:37.953809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:50473)
0928 23:20:37.953925 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:37.953961 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260928 23:20:38.788611 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:38.788334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:36675)
0928 23:20:38.788508 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:38.788556 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260928 23:20:38.843909 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:38.843721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:41533)
0928 23:20:38.843830 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:38.843866 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260928 23:20:38.954641 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:38.954442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:33283)
0928 23:20:38.954562 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:38.954600 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260928 23:20:39.789208 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:39.789005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:49035)
0928 23:20:39.789121 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:39.789159 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260928 23:20:39.844637 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:39.844369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:38685)
0928 23:20:39.844544 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:39.844592 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260928 23:20:39.955293 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:39.955084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:50329)
0928 23:20:39.955209 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:39.955248 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260928 23:20:40.789880 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:40.789673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.1:52343)
0928 23:20:40.789802 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:40.789837 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260928 23:20:40.845316 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:40.845115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.2:51585)
0928 23:20:40.845238 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:40.845274 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260928 23:20:40.956027 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:40.955810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38957 (local address 127.0.98.3:56431)
0928 23:20:40.955946 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:40.955983 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260928 23:20:40.960489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 16826
I20260928 23:20:40.964675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 16993
I20260928 23:20:40.968118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 17127
2026-09-28T23:20:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58135 ms)
[----------] 30 tests from SecurityITest (223337 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:41 dist-test-slave-djkr krb5kdc[17314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:41 dist-test-slave-djkr krb5kdc[17314](info): set up 2 sockets
Sep 28 23:20:41 dist-test-slave-djkr krb5kdc[17314](info): commencing operation
krb5kdc: starting...
W20260928 23:20:41.906961 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:41 dist-test-slave-djkr krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:20:42.019158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40761
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:42.096531 17327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:42.096729 17327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:42.096750 17327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:42.097819 17327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:42.097854 17327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:42.097867 17327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:42.097877 17327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:42.097887 17327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:42.099258 17327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:42.099445 17327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:42.099653 17327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:42.104228 17332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:42.104244 17335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:42.104332 17333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:42.104379 17327 server_base.cc:1034] running on GCE node
I20260928 23:20:42.104722 17327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:42.104996 17327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:42.106153 17327 hybrid_clock.cc:648] HybridClock initialized: now 1790637642106136 us; error 36 us; skew 500 ppm
Sep 28 23:20:42 dist-test-slave-djkr krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:42.108630 17327 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:20:42.109019 17327 webserver.cc:460] Webserver started at http://127.0.98.62:46011/ using document root <none> and password file <none>
I20260928 23:20:42.109221 17327 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:42.109267 17327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:42.109386 17327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:42.110385 17327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "9c72b2b71db243298cc360717a1c213a"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "087a5f6aaeddc1cd5f8c391fb8cc6414"
server_key_iv: "090cbdd80cd9df016d0d4b79513ef65a"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.110710 17327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c72b2b71db243298cc360717a1c213a"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "087a5f6aaeddc1cd5f8c391fb8cc6414"
server_key_iv: "090cbdd80cd9df016d0d4b79513ef65a"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.111929 17327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:20:42.112529 17342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:42.112685 17327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:42.112763 17327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "9c72b2b71db243298cc360717a1c213a"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "087a5f6aaeddc1cd5f8c391fb8cc6414"
server_key_iv: "090cbdd80cd9df016d0d4b79513ef65a"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.112821 17327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:42.119649 17327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:42.121346 17327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:42.121465 17327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:42.125077 17327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40761
I20260928 23:20:42.125126 17394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40761 every 8 connection(s)
I20260928 23:20:42.125422 17327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:42.126250 17395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:42.128445 17395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a: Bootstrap starting.
I20260928 23:20:42.128947 17395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:42.129180 17395 log.cc:824] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:42.129779 17395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a: No bootstrap required, opened a new log
I20260928 23:20:42.131294 17395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c72b2b71db243298cc360717a1c213a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40761 } }
I20260928 23:20:42.131434 17395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:42.131455 17395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c72b2b71db243298cc360717a1c213a, State: Initialized, Role: FOLLOWER
I20260928 23:20:42.131589 17395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c72b2b71db243298cc360717a1c213a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40761 } }
I20260928 23:20:42.131670 17395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:42.131698 17395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:42.131729 17395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:42.132215 17395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c72b2b71db243298cc360717a1c213a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40761 } }
I20260928 23:20:42.132300 17395 leader_election.cc:302] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c72b2b71db243298cc360717a1c213a; no voters:
I20260928 23:20:42.132414 17395 leader_election.cc:288] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:42.132464 17398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:42.132625 17395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:42.132655 17398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [term 1 LEADER]: Becoming Leader. State: Replica: 9c72b2b71db243298cc360717a1c213a, State: Running, Role: LEADER
I20260928 23:20:42.132804 17398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c72b2b71db243298cc360717a1c213a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40761 } }
I20260928 23:20:42.133145 17399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c72b2b71db243298cc360717a1c213a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c72b2b71db243298cc360717a1c213a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40761 } } }
I20260928 23:20:42.133283 17400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c72b2b71db243298cc360717a1c213a. Latest consensus state: current_term: 1 leader_uuid: "9c72b2b71db243298cc360717a1c213a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c72b2b71db243298cc360717a1c213a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40761 } } }
I20260928 23:20:42.133292 17399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:42.133380 17400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:42.133467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 17327
I20260928 23:20:42.133586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:42.133790 392 external_mini_cluster.cc:1428] Setting key 2250754084f7ebe775a6133592e64e3e
W20260928 23:20:42.134361 17413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:20:42.134460 17413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:20:42.134567 17407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:42.134717 17407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:42.136209 17407 catalog_manager.cc:1347] Generated new cluster ID: 887c2d722ac2413f894d7bd5a0ca60b9
I20260928 23:20:42.136261 17407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:20:42 dist-test-slave-djkr krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:42.143656 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.134240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.0.98.62:40761)
0928 23:20:42.134579 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.134589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.134619 (+ 30us) server_negotiation.cc:407] Connection header received
0928 23:20:42.134732 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.134739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.134793 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.134926 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:42.135629 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.136746 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.137471 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.137897 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.137989 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.140265 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:42.140269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:42.140269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:42.140280 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:42.142255 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:42.142572 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:42.142575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:42.142575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:42.142663 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:42.142840 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:42.142840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:42.142841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:42.143058 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:42.143203 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.143303 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.143433 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":66,"threads_started":1}
I20260928 23:20:42.145690 17407 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:42.146159 17407 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:42.151403 17407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a: Generated new TSK 0
I20260928 23:20:42.151562 17407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:20:42.178455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40761
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:42.279578 17421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:42.279799 17421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:42.279831 17421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:42.280988 17421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:42.281026 17421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:42.281061 17421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:20:42.282524 17421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:42.282755 17421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:42.282960 17421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:42.287794 17427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:42.287806 17426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:42.287801 17429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:42.288105 17421 server_base.cc:1034] running on GCE node
I20260928 23:20:42.288316 17421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:42.288528 17421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:42.289677 17421 hybrid_clock.cc:648] HybridClock initialized: now 1790637642289666 us; error 31 us; skew 500 ppm
Sep 28 23:20:42 dist-test-slave-djkr krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:42.292212 17421 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:20:42.292604 17421 webserver.cc:460] Webserver started at http://127.0.98.1:46331/ using document root <none> and password file <none>
I20260928 23:20:42.292795 17421 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:42.292842 17421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:42.292963 17421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:42.293990 17421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "efadeb6f12d846bd8bc0ce039aa73b05"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "2c2f204e726108cd5f4549c76ae581dc"
server_key_iv: "f6d1c2076a5d7c1a2d809cfe6ea36d37"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.294325 17421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "efadeb6f12d846bd8bc0ce039aa73b05"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "2c2f204e726108cd5f4549c76ae581dc"
server_key_iv: "f6d1c2076a5d7c1a2d809cfe6ea36d37"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.295611 17421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:20:42.296366 17436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:42.296571 17421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:42.296649 17421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "efadeb6f12d846bd8bc0ce039aa73b05"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "2c2f204e726108cd5f4549c76ae581dc"
server_key_iv: "f6d1c2076a5d7c1a2d809cfe6ea36d37"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.296716 17421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:42.319401 17421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:42.321209 17421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:42.321347 17421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:42.321566 17421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:42.321861 17421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:42.321892 17421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:42.321921 17421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:42.321938 17421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:42.327191 17421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43443
I20260928 23:20:42.327258 17549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43443 every 8 connection(s)
I20260928 23:20:42.327534 17421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:20:42 dist-test-slave-djkr krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:42.333678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 17421
I20260928 23:20:42.333778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:20:42.333999 392 external_mini_cluster.cc:1428] Setting key 06050a64584b22e7756f63ed40cfabf6
I20260928 23:20:42.336598 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.328253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49887 (local address 127.0.98.62:40761)
0928 23:20:42.328338 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.328341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.328513 (+ 172us) server_negotiation.cc:407] Connection header received
0928 23:20:42.329398 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.329399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.329414 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.329460 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:42.330021 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.330529 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.331480 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.331721 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.331774 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.334041 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:42.334042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:42.334043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:42.334051 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:42.335317 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:42.335736 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:42.335737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:42.335737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:42.335797 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:42.336012 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:42.336013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:42.336013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:42.336118 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:42.336173 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.336429 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.336514 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:20:42.336747 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.328233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40761 (local address 127.0.98.1:49887)
0928 23:20:42.328397 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:42.328432 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:42.329311 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:42.329523 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:42.329527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:42.329771 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:42.329952 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.329956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.330596 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.330596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.331389 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.331391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.331782 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.331783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.331879 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.332299 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:42.332306 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:42.333954 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:42.335395 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:42.335402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:42.335404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:42.335665 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:42.335852 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:42.335853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:42.335853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:42.335955 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:42.336172 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:42.336178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:42.336311 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 23:20:42.336397 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:20:42.336538 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 23:20:42.337160 17550 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40761
I20260928 23:20:42.337248 17550 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:42.337409 17550 heartbeater.cc:511] Master 127.0.98.62:40761 requested a full tablet report, sending...
I20260928 23:20:42.337901 17359 ts_manager.cc:194] Registered new tserver with Master: efadeb6f12d846bd8bc0ce039aa73b05 (127.0.98.1:43443)
I20260928 23:20:42.338495 17359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49887
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:20:42.356035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40761
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:42.433559 17555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:42.433756 17555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:42.433786 17555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:42.434947 17555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:42.434986 17555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:42.435014 17555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:20:42.436537 17555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:42.436758 17555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:42.436959 17555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:42.441591 17563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:42.441663 17560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:42.441610 17561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:42.441740 17555 server_base.cc:1034] running on GCE node
I20260928 23:20:42.441996 17555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:42.442216 17555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:42.443354 17555 hybrid_clock.cc:648] HybridClock initialized: now 1790637642443334 us; error 34 us; skew 500 ppm
Sep 28 23:20:42 dist-test-slave-djkr krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:42.445744 17555 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:20:42.446151 17555 webserver.cc:460] Webserver started at http://127.0.98.2:42505/ using document root <none> and password file <none>
I20260928 23:20:42.446337 17555 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:42.446379 17555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:42.446506 17555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:42.447613 17555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "7750739e5cef4a43b58710d5a16a2f19"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "21aab664806b0559c3a2bb7e07cbf3ca"
server_key_iv: "9bf68556674be2990c93f51eff3b8658"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.447937 17555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7750739e5cef4a43b58710d5a16a2f19"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "21aab664806b0559c3a2bb7e07cbf3ca"
server_key_iv: "9bf68556674be2990c93f51eff3b8658"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.449066 17555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:20:42.449705 17570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:42.449846 17555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:20:42.449915 17555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "7750739e5cef4a43b58710d5a16a2f19"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "21aab664806b0559c3a2bb7e07cbf3ca"
server_key_iv: "9bf68556674be2990c93f51eff3b8658"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.449978 17555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:42.474741 17555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:42.476701 17555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:42.476819 17555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:42.477030 17555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:42.477301 17555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:42.477334 17555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:42.477355 17555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:42.477372 17555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:42.482451 17555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42789
I20260928 23:20:42.482524 17683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42789 every 8 connection(s)
I20260928 23:20:42.482776 17555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:20:42 dist-test-slave-djkr krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:42.490302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 17555
I20260928 23:20:42.490398 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:20:42.490567 392 external_mini_cluster.cc:1428] Setting key 0b809c4eaa412f73e98891542de1d9e0
I20260928 23:20:42.491593 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.483429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53477 (local address 127.0.98.62:40761)
0928 23:20:42.483493 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.483495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.483693 (+ 198us) server_negotiation.cc:407] Connection header received
0928 23:20:42.484693 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.484694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.484704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.484757 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:42.485302 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.485789 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.486770 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.486956 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.486997 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.489080 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:42.489081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:42.489082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:42.489088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:42.490232 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:42.490653 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:42.490654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:42.490654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:42.490717 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:42.490946 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:42.490946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:42.490946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:42.491068 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:42.491133 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.491397 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.491474 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:20:42.491736 17686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.483390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40761 (local address 127.0.98.2:53477)
0928 23:20:42.483538 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:42.483606 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:42.484600 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:42.484814 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:42.484818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:42.485051 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:42.485240 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.485244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.485856 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.485857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.486699 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.486702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.487008 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.487009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.487112 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.487633 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:42.487642 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:42.488974 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:42.490313 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:42.490319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:42.490321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:42.490546 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:42.490785 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:42.490786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:42.490786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:42.490902 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:42.491135 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:42.491140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:42.491275 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 23:20:42.491362 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:20:42.491503 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260928 23:20:42.492877 17684 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40761
I20260928 23:20:42.492970 17684 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:42.493139 17684 heartbeater.cc:511] Master 127.0.98.62:40761 requested a full tablet report, sending...
I20260928 23:20:42.493491 17359 ts_manager.cc:194] Registered new tserver with Master: 7750739e5cef4a43b58710d5a16a2f19 (127.0.98.2:42789)
I20260928 23:20:42.493837 17359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53477
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:20:42.514474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40761
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:42.595618 17689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:42.596022 17689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:42.596055 17689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:42.597743 17689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:42.597832 17689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:42.597882 17689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:20:42.599716 17689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:42.600119 17689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:42.600399 17689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:42.606426 17697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:42.606441 17695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:42.606441 17694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:42.606561 17689 server_base.cc:1034] running on GCE node
I20260928 23:20:42.606899 17689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:42.607127 17689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:42.608285 17689 hybrid_clock.cc:648] HybridClock initialized: now 1790637642608268 us; error 30 us; skew 500 ppm
Sep 28 23:20:42 dist-test-slave-djkr krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:42.610714 17689 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:20:42.611198 17689 webserver.cc:460] Webserver started at http://127.0.98.3:45507/ using document root <none> and password file <none>
I20260928 23:20:42.611423 17689 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:42.611469 17689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:42.611624 17689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:42.612969 17689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "729312f9f80c473fa62cbea613ad46a1"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "70528ed63a6f53c41557e9f6a27bb669"
server_key_iv: "b3082fef6ba2602938b5f5e96f579ffb"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.613380 17689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "729312f9f80c473fa62cbea613ad46a1"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "70528ed63a6f53c41557e9f6a27bb669"
server_key_iv: "b3082fef6ba2602938b5f5e96f579ffb"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.614948 17689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:20:42.615818 17704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:42.615998 17689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:42.616079 17689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "729312f9f80c473fa62cbea613ad46a1"
format_stamp: "Formatted at 2026-09-28 23:20:42 on dist-test-slave-djkr"
server_key: "70528ed63a6f53c41557e9f6a27bb669"
server_key_iv: "b3082fef6ba2602938b5f5e96f579ffb"
server_key_version: "encryptionkey@0"
I20260928 23:20:42.616158 17689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:42.629819 17689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:42.632189 17689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:42.632325 17689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:42.632558 17689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:42.632918 17689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:42.632963 17689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:42.632995 17689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:42.633020 17689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:42.638428 17689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36705
I20260928 23:20:42.638511 17817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36705 every 8 connection(s)
I20260928 23:20:42.638765 17689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:20:42.638985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 17689
I20260928 23:20:42.639089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:20:42.639287 392 external_mini_cluster.cc:1428] Setting key 5a78a4fc104579ee3f7dc3dc88519c43
Sep 28 23:20:42 dist-test-slave-djkr krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:42.647710 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.639538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52131 (local address 127.0.98.62:40761)
0928 23:20:42.639609 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.639611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.639860 (+ 249us) server_negotiation.cc:407] Connection header received
0928 23:20:42.640828 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.640829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.640839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.640877 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:42.641430 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.641920 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.642950 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.643144 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.643198 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.645259 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:42.645260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:42.645260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:42.645267 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:42.646475 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:42.646815 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:42.646816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:42.646816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:42.646863 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:42.647153 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:42.647154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:42.647154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:42.647256 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:42.647307 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.647575 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.647648 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:20:42.647887 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.639515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40761 (local address 127.0.98.3:52131)
0928 23:20:42.639726 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:42.639772 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:42.640745 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:42.640936 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:42.640939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:42.641177 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:42.641352 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.641356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.642034 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.642035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.642859 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.642861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.643219 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.643220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.643320 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.643851 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:42.643859 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:42.645168 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:42.646555 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:42.646561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:42.646563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:42.646754 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:42.646943 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:42.646944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:42.646944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:42.647068 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:42.647329 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:42.647335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:42.647449 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 23:20:42.647537 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:20:42.647687 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":133,"threads_started":1}
I20260928 23:20:42.648304 17818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40761
I20260928 23:20:42.648384 17818 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:42.648547 17818 heartbeater.cc:511] Master 127.0.98.62:40761 requested a full tablet report, sending...
I20260928 23:20:42.648864 17359 ts_manager.cc:194] Registered new tserver with Master: 729312f9f80c473fa62cbea613ad46a1 (127.0.98.3:36705)
I20260928 23:20:42.649204 17359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52131
I20260928 23:20:42.650892 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:20:42.657624 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.652172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.0.98.62:40761)
0928 23:20:42.652206 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.652207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.652290 (+ 83us) server_negotiation.cc:407] Connection header received
0928 23:20:42.652338 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.652338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.652345 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.652387 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:42.652902 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.653358 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.654143 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.654317 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.654365 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.655436 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:42.655437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:42.655437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:42.655443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:42.656629 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:42.656972 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:42.656973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:42.656973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:42.657041 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:42.657224 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:42.657225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:42.657225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:42.657351 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:42.657418 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.657507 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.657565 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:20:42.659093 17359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:20:42.660641 17359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:20:42.666658 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.662659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43443 (local address 127.0.0.1:43894)
0928 23:20:42.662892 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:42.662904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:42.662954 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:42.663382 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:42.663383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:42.663385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:42.663415 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.663416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.664389 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.664390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.665768 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.665770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.666388 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.666389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.666441 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.666459 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:20:42.666519 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 23:20:42.666576 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":32,"thread_start_us":54,"threads_started":1}
I20260928 23:20:42.666682 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.663045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.0.98.2:42789)
0928 23:20:42.663224 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.663228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.663238 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:20:42.663274 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.663275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.663289 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.663360 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:42.663594 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.664655 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.665562 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.666271 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.666346 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.666403 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.666569 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.666607 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260928 23:20:42.666797 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.662822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.0.98.3:36705)
0928 23:20:42.663046 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.663050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.663059 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:20:42.663088 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.663089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.663116 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.663190 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:42.663343 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.664554 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.665802 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.666463 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.666518 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.666571 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.666688 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.666724 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
I20260928 23:20:42.666775 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.662659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.0.1:41692)
0928 23:20:42.662822 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:42.662860 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:42.662928 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:42.663200 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:42.663203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:42.663205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:42.663270 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.663272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.664663 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.664664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.665718 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.665721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.666538 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.666538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.666612 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.666617 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:20:42.666665 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:20:42.666706 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260928 23:20:42.666852 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.662676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42789 (local address 127.0.0.1:35396)
0928 23:20:42.663024 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:42.663032 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:42.663077 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:42.663446 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:42.663447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:42.663450 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:42.663496 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.663498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.664732 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.664733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.665478 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.665481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.666362 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.666363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.666442 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.666459 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 23:20:42.666524 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 23:20:42.666800 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":234,"thread_start_us":77,"threads_started":1}
I20260928 23:20:42.667382 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.662840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.0.98.1:43443)
0928 23:20:42.663241 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.663245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.663255 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:20:42.663292 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.663294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.663308 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.663387 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:42.663481 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.664298 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.665844 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.666303 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.667073 (+ 770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.667238 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.667270 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.667311 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":83,"threads_started":1}
I20260928 23:20:42.667744 17752 tablet_service.cc:1467] Processing CreateTablet for tablet da9e10d34f6941d3b553f218c99530ff (DEFAULT_TABLE table=test-table [id=ab76a5f46e634c83a0bd61918a156e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:42.667742 17618 tablet_service.cc:1467] Processing CreateTablet for tablet da9e10d34f6941d3b553f218c99530ff (DEFAULT_TABLE table=test-table [id=ab76a5f46e634c83a0bd61918a156e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:42.668061 17752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9e10d34f6941d3b553f218c99530ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:42.667972 17484 tablet_service.cc:1467] Processing CreateTablet for tablet da9e10d34f6941d3b553f218c99530ff (DEFAULT_TABLE table=test-table [id=ab76a5f46e634c83a0bd61918a156e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:42.668072 17618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9e10d34f6941d3b553f218c99530ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:42.668236 17484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9e10d34f6941d3b553f218c99530ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:42.670557 17835 tablet_bootstrap.cc:492] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: Bootstrap starting.
I20260928 23:20:42.671113 17835 tablet_bootstrap.cc:654] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:42.671268 17836 tablet_bootstrap.cc:492] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05: Bootstrap starting.
I20260928 23:20:42.671715 17837 tablet_bootstrap.cc:492] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: Bootstrap starting.
I20260928 23:20:42.671729 17835 log.cc:824] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:42.671947 17836 tablet_bootstrap.cc:654] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:42.672293 17837 tablet_bootstrap.cc:654] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:42.672494 17836 log.cc:824] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:42.672797 17837 log.cc:824] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:42.672997 17835 tablet_bootstrap.cc:492] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: No bootstrap required, opened a new log
I20260928 23:20:42.673056 17835 ts_tablet_manager.cc:1397] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:20:42.673158 17836 tablet_bootstrap.cc:492] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05: No bootstrap required, opened a new log
I20260928 23:20:42.673205 17836 ts_tablet_manager.cc:1397] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:20:42.673418 17837 tablet_bootstrap.cc:492] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: No bootstrap required, opened a new log
I20260928 23:20:42.673470 17837 ts_tablet_manager.cc:1397] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:20:42.674933 17835 raft_consensus.cc:348] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43443 } } peers { permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260928 23:20:42.674973 17836 raft_consensus.cc:348] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43443 } } peers { permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260928 23:20:42.675104 17835 raft_consensus.cc:374] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:42.675114 17836 raft_consensus.cc:374] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:42.675139 17835 raft_consensus.cc:729] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729312f9f80c473fa62cbea613ad46a1, State: Initialized, Role: FOLLOWER
I20260928 23:20:42.675145 17836 raft_consensus.cc:729] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efadeb6f12d846bd8bc0ce039aa73b05, State: Initialized, Role: FOLLOWER
I20260928 23:20:42.675251 17835 consensus_queue.cc:260] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43443 } } peers { permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260928 23:20:42.675249 17836 consensus_queue.cc:260] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43443 } } peers { permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260928 23:20:42.675508 17835 ts_tablet_manager.cc:1428] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:20:42.675508 17836 ts_tablet_manager.cc:1428] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 23:20:42.675678 17818 heartbeater.cc:503] Master 127.0.98.62:40761 was elected leader, sending a full tablet report...
I20260928 23:20:42.675690 17550 heartbeater.cc:503] Master 127.0.98.62:40761 was elected leader, sending a full tablet report...
I20260928 23:20:42.675786 17837 raft_consensus.cc:348] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43443 } } peers { permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260928 23:20:42.675946 17837 raft_consensus.cc:374] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:42.675981 17837 raft_consensus.cc:729] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7750739e5cef4a43b58710d5a16a2f19, State: Initialized, Role: FOLLOWER
I20260928 23:20:42.676083 17837 consensus_queue.cc:260] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43443 } } peers { permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260928 23:20:42.676389 17684 heartbeater.cc:503] Master 127.0.98.62:40761 was elected leader, sending a full tablet report...
I20260928 23:20:42.676425 17837 ts_tablet_manager.cc:1428] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260928 23:20:42.733387 17685 tablet.cc:2369] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:20:42.828255 17551 tablet.cc:2369] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:20:42.852445 17842 raft_consensus.cc:482] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:42.852620 17842 raft_consensus.cc:504] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43443 } } peers { permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260928 23:20:42.852969 17842 leader_election.cc:288] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 729312f9f80c473fa62cbea613ad46a1 (127.0.98.3:36705), 7750739e5cef4a43b58710d5a16a2f19 (127.0.98.2:42789)
I20260928 23:20:42.856879 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.853173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52905 (local address 127.0.98.2:42789)
0928 23:20:42.853218 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.853221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.853391 (+ 170us) server_negotiation.cc:407] Connection header received
0928 23:20:42.853438 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.853438 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.853450 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.853498 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:42.853654 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.854360 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.855884 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.856463 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.856547 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.856559 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.856764 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.856812 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:20:42.856879 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.853225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46923 (local address 127.0.98.3:36705)
0928 23:20:42.853269 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.853271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.853330 (+ 59us) server_negotiation.cc:407] Connection header received
0928 23:20:42.853395 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.853396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.853404 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.853442 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:42.853604 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.854349 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.855883 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.856463 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.856567 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.856576 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.856764 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.856812 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:20:42.856895 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.853130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.98.1:46923)
0928 23:20:42.853265 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:42.853274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:42.853333 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:42.853453 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:42.853454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:42.853456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:42.853530 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.853532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.854436 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.854437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.855787 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.855790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.856575 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.856576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.856656 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.856661 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:20:42.856720 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:20:42.856773 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":55,"thread_start_us":75,"threads_started":1}
I20260928 23:20:42.856895 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.853141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42789 (local address 127.0.98.1:52905)
0928 23:20:42.853328 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:42.853336 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:42.853387 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:42.853512 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:42.853512 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:42.853514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:42.853559 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.853560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.854436 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.854437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.855787 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:42.855790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.856553 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.856554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:42.856665 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.856667 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:20:42.856720 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:20:42.856773 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":44,"thread_start_us":69,"threads_started":1}
I20260928 23:20:42.857236 17772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9e10d34f6941d3b553f218c99530ff" candidate_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729312f9f80c473fa62cbea613ad46a1" is_pre_election: true
I20260928 23:20:42.857390 17772 raft_consensus.cc:2454] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efadeb6f12d846bd8bc0ce039aa73b05 in term 0.
I20260928 23:20:42.857473 17638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9e10d34f6941d3b553f218c99530ff" candidate_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7750739e5cef4a43b58710d5a16a2f19" is_pre_election: true
I20260928 23:20:42.857616 17638 raft_consensus.cc:2454] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efadeb6f12d846bd8bc0ce039aa73b05 in term 0.
I20260928 23:20:42.857609 17438 leader_election.cc:302] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729312f9f80c473fa62cbea613ad46a1, efadeb6f12d846bd8bc0ce039aa73b05; no voters:
I20260928 23:20:42.857740 17842 raft_consensus.cc:2781] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:20:42.857807 17842 raft_consensus.cc:482] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:20:42.857829 17842 raft_consensus.cc:3037] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:42.858471 17842 raft_consensus.cc:504] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43443 } } peers { permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260928 23:20:42.858577 17842 leader_election.cc:288] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [CANDIDATE]: Term 1 election: Requested vote from peers 729312f9f80c473fa62cbea613ad46a1 (127.0.98.3:36705), 7750739e5cef4a43b58710d5a16a2f19 (127.0.98.2:42789)
I20260928 23:20:42.858731 17772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9e10d34f6941d3b553f218c99530ff" candidate_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "729312f9f80c473fa62cbea613ad46a1"
I20260928 23:20:42.858748 17638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9e10d34f6941d3b553f218c99530ff" candidate_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7750739e5cef4a43b58710d5a16a2f19"
I20260928 23:20:42.858804 17638 raft_consensus.cc:3037] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:42.858808 17772 raft_consensus.cc:3037] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:42.859493 17772 raft_consensus.cc:2454] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efadeb6f12d846bd8bc0ce039aa73b05 in term 1.
I20260928 23:20:42.859602 17638 raft_consensus.cc:2454] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efadeb6f12d846bd8bc0ce039aa73b05 in term 1.
I20260928 23:20:42.859784 17438 leader_election.cc:302] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729312f9f80c473fa62cbea613ad46a1, efadeb6f12d846bd8bc0ce039aa73b05; no voters:
I20260928 23:20:42.859894 17842 raft_consensus.cc:2781] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:42.860024 17842 raft_consensus.cc:686] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [term 1 LEADER]: Becoming Leader. State: Replica: efadeb6f12d846bd8bc0ce039aa73b05, State: Running, Role: LEADER
I20260928 23:20:42.860111 17842 consensus_queue.cc:237] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43443 } } peers { permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260928 23:20:42.860910 17357 catalog_manager.cc:5495] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to efadeb6f12d846bd8bc0ce039aa73b05 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efadeb6f12d846bd8bc0ce039aa73b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } health_report { overall_health: UNKNOWN } } }
W20260928 23:20:42.889261 17819 tablet.cc:2369] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:20:42.894609 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.892376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.0.98.1:43443)
0928 23:20:42.892433 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:42.892435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:42.892443 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:20:42.892484 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:42.892484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:42.892502 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:42.892560 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:20:42.892690 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.893144 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.893938 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:42.894121 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:42.894173 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:42.894333 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:20:42.894427 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:20:42.894494 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:42.894518 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 23:20:42.894540 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:20:42.896236 17638 raft_consensus.cc:1261] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19 [term 1 FOLLOWER]: Refusing update from remote peer efadeb6f12d846bd8bc0ce039aa73b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:42.896232 17772 raft_consensus.cc:1261] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1 [term 1 FOLLOWER]: Refusing update from remote peer efadeb6f12d846bd8bc0ce039aa73b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:42.896579 17842 consensus_queue.cc:1035] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7750739e5cef4a43b58710d5a16a2f19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:42.896725 17846 consensus_queue.cc:1035] T da9e10d34f6941d3b553f218c99530ff P efadeb6f12d846bd8bc0ce039aa73b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "729312f9f80c473fa62cbea613ad46a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:42.899853 17850 mvcc.cc:205] Tried to move back new op lower bound from 7334451785300606976 to 7334451785155579904. Current Snapshot: MvccSnapshot[applied={T|T < 7334451785300606976}]
I20260928 23:20:42.900084 17849 mvcc.cc:205] Tried to move back new op lower bound from 7334451785300606976 to 7334451785155579904. Current Snapshot: MvccSnapshot[applied={T|T < 7334451785300606976 or (T in {7334451785300606976})}]
I20260928 23:20:42.901099 17357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:42.901211 17357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:42.901878 17357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c72b2b71db243298cc360717a1c213a: Sending DeleteTablet for 3 replicas of tablet da9e10d34f6941d3b553f218c99530ff
I20260928 23:20:42.902150 17484 tablet_service.cc:1514] Processing DeleteTablet for tablet da9e10d34f6941d3b553f218c99530ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43894
I20260928 23:20:42.902184 17618 tablet_service.cc:1514] Processing DeleteTablet for tablet da9e10d34f6941d3b553f218c99530ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35396
I20260928 23:20:42.902424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 17421
I20260928 23:20:42.903054 17752 tablet_service.cc:1514] Processing DeleteTablet for tablet da9e10d34f6941d3b553f218c99530ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41692
I20260928 23:20:42.903245 17861 tablet_replica.cc:330] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: stopping tablet replica
I20260928 23:20:42.903313 17860 tablet_replica.cc:330] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: stopping tablet replica
I20260928 23:20:42.903331 17861 raft_consensus.cc:2229] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:42.903384 17860 raft_consensus.cc:2229] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:42.903419 17861 raft_consensus.cc:2258] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:42.903460 17860 raft_consensus.cc:2258] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:42.903847 17860 ts_tablet_manager.cc:1905] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:42.903856 17861 ts_tablet_manager.cc:1905] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:42.905452 17861 ts_tablet_manager.cc:1918] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:42.905514 17861 log.cc:1196] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/da9e10d34f6941d3b553f218c99530ff
I20260928 23:20:42.905622 17860 ts_tablet_manager.cc:1918] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:42.905710 17860 log.cc:1196] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/da9e10d34f6941d3b553f218c99530ff
I20260928 23:20:42.905754 17861 ts_tablet_manager.cc:1939] T da9e10d34f6941d3b553f218c99530ff P 729312f9f80c473fa62cbea613ad46a1: Deleting consensus metadata
I20260928 23:20:42.905944 17860 ts_tablet_manager.cc:1939] T da9e10d34f6941d3b553f218c99530ff P 7750739e5cef4a43b58710d5a16a2f19: Deleting consensus metadata
I20260928 23:20:42.906287 17346 catalog_manager.cc:4841] TS 7750739e5cef4a43b58710d5a16a2f19 (127.0.98.2:42789): tablet da9e10d34f6941d3b553f218c99530ff (table test-table [id=ab76a5f46e634c83a0bd61918a156e32]) successfully deleted
I20260928 23:20:42.906339 17344 catalog_manager.cc:4841] TS 729312f9f80c473fa62cbea613ad46a1 (127.0.98.3:36705): tablet da9e10d34f6941d3b553f218c99530ff (table test-table [id=ab76a5f46e634c83a0bd61918a156e32]) successfully deleted
W20260928 23:20:42.907351 17343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43443 (error 108)
I20260928 23:20:42.907527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 17555
I20260928 23:20:42.908176 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:42.908070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43443 (local address 127.0.0.1:43902)
0928 23:20:42.908112 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:42.908145 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:20:42.908281 17343 catalog_manager.cc:4584] TS efadeb6f12d846bd8bc0ce039aa73b05 (127.0.98.1:43443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da9e10d34f6941d3b553f218c99530ff: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43443: connect: Connection refused (error 111)
I20260928 23:20:42.911995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 17689
I20260928 23:20:42.916023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 17327
2026-09-28T23:20:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1947 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T23:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:42Z Disabled control of system clock
I20260928 23:20:42.940761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41573
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:20:43.017243 17866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:43.017442 17866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:43.017467 17866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:43.018595 17866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:43.018642 17866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:43.018653 17866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:43.018664 17866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:43.018683 17866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:43.019992 17866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41573
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:43.020215 17866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:43.020473 17866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:43.024813 17874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:43.024904 17872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:43.024986 17866 server_base.cc:1034] running on GCE node
W20260928 23:20:43.024864 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:43.025256 17866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:43.025516 17866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:43.026670 17866 hybrid_clock.cc:648] HybridClock initialized: now 1790637643026653 us; error 29 us; skew 500 ppm
I20260928 23:20:43.027863 17866 webserver.cc:460] Webserver started at http://127.0.98.62:34265/ using document root <none> and password file <none>
I20260928 23:20:43.028074 17866 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:43.028120 17866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:43.028239 17866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:43.029220 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "d97c8ad34f2845db82a2b5402ffefe9f"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "6ec0f00d3918d71fa09d4b542d20d8f1"
server_key_iv: "3533809b349cebcb40d3c25bb880d8dc"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.029541 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "d97c8ad34f2845db82a2b5402ffefe9f"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "6ec0f00d3918d71fa09d4b542d20d8f1"
server_key_iv: "3533809b349cebcb40d3c25bb880d8dc"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.030732 17866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:20:43.031401 17880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.031579 17866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:20:43.031692 17866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "d97c8ad34f2845db82a2b5402ffefe9f"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "6ec0f00d3918d71fa09d4b542d20d8f1"
server_key_iv: "3533809b349cebcb40d3c25bb880d8dc"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.031759 17866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:43.053359 17866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:43.053660 17866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:43.053788 17866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:43.057551 17866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41573
I20260928 23:20:43.057585 17932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41573 every 8 connection(s)
I20260928 23:20:43.057896 17866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:43.058770 17933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:43.060950 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f: Bootstrap starting.
I20260928 23:20:43.061439 17933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:43.061657 17933 log.cc:824] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:43.062217 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f: No bootstrap required, opened a new log
I20260928 23:20:43.063748 17933 raft_consensus.cc:348] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c8ad34f2845db82a2b5402ffefe9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41573 } }
I20260928 23:20:43.063879 17933 raft_consensus.cc:374] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:43.063898 17933 raft_consensus.cc:729] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97c8ad34f2845db82a2b5402ffefe9f, State: Initialized, Role: FOLLOWER
I20260928 23:20:43.064002 17933 consensus_queue.cc:260] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c8ad34f2845db82a2b5402ffefe9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41573 } }
I20260928 23:20:43.064072 17933 raft_consensus.cc:388] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:43.064111 17933 raft_consensus.cc:482] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:43.064160 17933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:43.064258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 17866
I20260928 23:20:43.064359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:43.064563 392 external_mini_cluster.cc:1428] Setting key 44eada271332fd358ab7617e070af2db
I20260928 23:20:43.064779 17933 raft_consensus.cc:504] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c8ad34f2845db82a2b5402ffefe9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41573 } }
I20260928 23:20:43.064868 17933 leader_election.cc:302] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d97c8ad34f2845db82a2b5402ffefe9f; no voters:
I20260928 23:20:43.065013 17933 leader_election.cc:288] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:43.065092 17937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:43.065236 17933 sys_catalog.cc:564] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:43.065248 17937 raft_consensus.cc:686] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [term 1 LEADER]: Becoming Leader. State: Replica: d97c8ad34f2845db82a2b5402ffefe9f, State: Running, Role: LEADER
I20260928 23:20:43.065354 17937 consensus_queue.cc:237] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c8ad34f2845db82a2b5402ffefe9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41573 } }
I20260928 23:20:43.065722 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d97c8ad34f2845db82a2b5402ffefe9f. Latest consensus state: current_term: 1 leader_uuid: "d97c8ad34f2845db82a2b5402ffefe9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c8ad34f2845db82a2b5402ffefe9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41573 } } }
I20260928 23:20:43.065806 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:43.065897 17939 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d97c8ad34f2845db82a2b5402ffefe9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97c8ad34f2845db82a2b5402ffefe9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41573 } } }
I20260928 23:20:43.065994 17947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:43.065999 17939 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:43.066480 17947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:43.067978 17947 catalog_manager.cc:1347] Generated new cluster ID: cf4c9765874d433ba927d4898ec6a94b
I20260928 23:20:43.068028 17947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:43.070900 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.065002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.0.98.62:41573)
0928 23:20:43.065423 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.065432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.065465 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:20:43.065578 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.065585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.065658 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.065808 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:43.066294 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.067604 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.068688 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.069283 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.069734 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.070093 (+ 359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:43.070097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:43.070097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:43.070287 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:43.070329 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:43.070343 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:43.070581 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.070623 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.070704 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":77,"threads_started":1}
I20260928 23:20:43.081836 17947 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:43.082297 17947 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:43.086853 17947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f: Generated new TSK 0
I20260928 23:20:43.086999 17947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:43.090250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41573
--builtin_ntp_servers=127.0.98.20:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:20:43.167573 17957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:43.167768 17957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:43.167788 17957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:43.168879 17957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:43.168916 17957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:43.168939 17957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:20:43.170213 17957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:43.170403 17957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:43.170660 17957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:43.175302 17962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:43.175350 17963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:43.175447 17957 server_base.cc:1034] running on GCE node
W20260928 23:20:43.175606 17965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:43.175793 17957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:43.176055 17957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:43.177227 17957 hybrid_clock.cc:648] HybridClock initialized: now 1790637643177206 us; error 32 us; skew 500 ppm
I20260928 23:20:43.178619 17957 webserver.cc:460] Webserver started at http://127.0.98.1:43017/ using document root <none> and password file <none>
I20260928 23:20:43.178857 17957 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:43.178920 17957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:43.179040 17957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:43.180282 17957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "0c55ae24c3e24b51be84ea1970abd57a"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "dd0f2f1109c6e42676a44e3e2ae9544d"
server_key_iv: "912285254d8e8ad019f148367b56c0bc"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.180641 17957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0c55ae24c3e24b51be84ea1970abd57a"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "dd0f2f1109c6e42676a44e3e2ae9544d"
server_key_iv: "912285254d8e8ad019f148367b56c0bc"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.182025 17957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:20:43.182765 17971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.182904 17957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:43.182983 17957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "0c55ae24c3e24b51be84ea1970abd57a"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "dd0f2f1109c6e42676a44e3e2ae9544d"
server_key_iv: "912285254d8e8ad019f148367b56c0bc"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.183046 17957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:43.205996 17957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:43.206213 17957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:43.206303 17957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:43.206470 17957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:43.206713 17957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:43.206741 17957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.206760 17957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:43.206771 17957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.211825 17957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34431
I20260928 23:20:43.211864 18084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34431 every 8 connection(s)
I20260928 23:20:43.212141 17957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:20:43.214529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 17957
I20260928 23:20:43.214620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:20:43.214810 392 external_mini_cluster.cc:1428] Setting key f725053b23ecce0c5c8e641400c37e67
I20260928 23:20:43.215705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41573
--builtin_ntp_servers=127.0.98.20:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:20:43.217466 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.212761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36119 (local address 127.0.98.62:41573)
0928 23:20:43.212813 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.212815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.213020 (+ 205us) server_negotiation.cc:407] Connection header received
0928 23:20:43.213484 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.213485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.213498 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.213544 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:43.214207 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.214890 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.216173 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.216447 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.216508 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.217230 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:43.217231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:43.217231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:43.217236 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:43.217253 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:43.217258 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:43.217318 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.217365 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.217391 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:20:43.217610 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.212734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41573 (local address 127.0.98.1:36119)
0928 23:20:43.212915 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:43.212946 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:43.213392 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:43.213626 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:43.213630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:43.213638 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:43.214099 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.214104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.214990 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.214992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.216068 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.216071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.216514 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.216515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.216619 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.217087 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:20:43.217088 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:43.217127 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:20:43.217148 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:43.217319 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:43.217321 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:43.217323 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:20:43.217363 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 23:20:43.217434 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260928 23:20:43.218068 18085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41573
I20260928 23:20:43.218161 18085 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:43.218319 18085 heartbeater.cc:511] Master 127.0.98.62:41573 requested a full tablet report, sending...
I20260928 23:20:43.218804 17897 ts_manager.cc:194] Registered new tserver with Master: 0c55ae24c3e24b51be84ea1970abd57a (127.0.98.1:34431)
I20260928 23:20:43.219686 17897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36119
W20260928 23:20:43.293850 18088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:43.294040 18088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:43.294059 18088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:43.295131 18088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:43.295163 18088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:43.295188 18088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:20:43.296466 18088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:43.296639 18088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:43.296875 18088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:43.301532 18096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:43.301544 18093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:43.301604 18088 server_base.cc:1034] running on GCE node
W20260928 23:20:43.301532 18094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:43.302004 18088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:43.302232 18088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:43.303366 18088 hybrid_clock.cc:648] HybridClock initialized: now 1790637643303358 us; error 25 us; skew 500 ppm
I20260928 23:20:43.304512 18088 webserver.cc:460] Webserver started at http://127.0.98.2:34573/ using document root <none> and password file <none>
I20260928 23:20:43.304744 18088 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:43.304791 18088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:43.304914 18088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:43.305961 18088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "0dfc93e29e4447d0b95d0159550fc8d8"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "bdb4d45aa9c6e1ba9e727c6164869bad"
server_key_iv: "dffdc0296ae68df98c6c9dc7d5e98d3b"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.306285 18088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0dfc93e29e4447d0b95d0159550fc8d8"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "bdb4d45aa9c6e1ba9e727c6164869bad"
server_key_iv: "dffdc0296ae68df98c6c9dc7d5e98d3b"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.307467 18088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:20:43.308177 18102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.308320 18088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.308401 18088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "0dfc93e29e4447d0b95d0159550fc8d8"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "bdb4d45aa9c6e1ba9e727c6164869bad"
server_key_iv: "dffdc0296ae68df98c6c9dc7d5e98d3b"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.308467 18088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:43.327461 18088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:43.327765 18088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:43.327893 18088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:43.328106 18088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:43.328397 18088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:43.328430 18088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.328459 18088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:43.328475 18088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.333757 18088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36647
I20260928 23:20:43.333804 18215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36647 every 8 connection(s)
I20260928 23:20:43.334105 18088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:20:43.338754 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.334730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42743 (local address 127.0.98.62:41573)
0928 23:20:43.334788 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.334790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.335002 (+ 212us) server_negotiation.cc:407] Connection header received
0928 23:20:43.335530 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.335531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.335541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.335622 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:43.336241 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.336727 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.337627 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.337814 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.337857 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.338538 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:43.338540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:43.338540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:43.338544 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:43.338557 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:43.338561 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:43.338597 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.338660 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.338698 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:20:43.338909 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.334732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41573 (local address 127.0.98.2:42743)
0928 23:20:43.334901 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:43.334934 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:43.335442 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:43.335681 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:43.335684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:43.335693 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:43.336177 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.336181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.336791 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.336791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.337566 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.337568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.337869 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.337870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.337971 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.338424 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:20:43.338424 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:43.338452 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:20:43.338473 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:43.338604 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:43.338606 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:43.338609 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:43.338655 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 23:20:43.338729 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260928 23:20:43.339349 18216 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41573
I20260928 23:20:43.339423 18216 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:43.339618 18216 heartbeater.cc:511] Master 127.0.98.62:41573 requested a full tablet report, sending...
I20260928 23:20:43.339950 17897 ts_manager.cc:194] Registered new tserver with Master: 0dfc93e29e4447d0b95d0159550fc8d8 (127.0.98.2:36647)
I20260928 23:20:43.340306 17897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42743
I20260928 23:20:43.340318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 18088
I20260928 23:20:43.340399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:20:43.340572 392 external_mini_cluster.cc:1428] Setting key 979efe7083eccb90b458564b4eacb187
I20260928 23:20:43.341358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41573
--builtin_ntp_servers=127.0.98.20:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:20:43.418490 18219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:43.418676 18219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:43.418695 18219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:43.419796 18219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:43.419832 18219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:43.419855 18219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:20:43.421106 18219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:43.421294 18219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:43.421521 18219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:43.426075 18225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:43.426096 18219 server_base.cc:1034] running on GCE node
W20260928 23:20:43.426065 18227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:43.426065 18224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:43.426529 18219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:43.426753 18219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:43.427892 18219 hybrid_clock.cc:648] HybridClock initialized: now 1790637643427851 us; error 51 us; skew 500 ppm
I20260928 23:20:43.429023 18219 webserver.cc:460] Webserver started at http://127.0.98.3:46831/ using document root <none> and password file <none>
I20260928 23:20:43.429251 18219 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:43.429301 18219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:43.429419 18219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:43.430421 18219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "45c932feecc44965ba5d450fc0f7ef29"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "2173d92f363e9e75d407e48740996622"
server_key_iv: "5c7cd45fbecbdfdb3b3d2a3603f207af"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.430742 18219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "45c932feecc44965ba5d450fc0f7ef29"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "2173d92f363e9e75d407e48740996622"
server_key_iv: "5c7cd45fbecbdfdb3b3d2a3603f207af"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.431934 18219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:20:43.432572 18233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.432721 18219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:43.432797 18219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "45c932feecc44965ba5d450fc0f7ef29"
format_stamp: "Formatted at 2026-09-28 23:20:43 on dist-test-slave-djkr"
server_key: "2173d92f363e9e75d407e48740996622"
server_key_iv: "5c7cd45fbecbdfdb3b3d2a3603f207af"
server_key_version: "encryptionkey@0"
I20260928 23:20:43.432854 18219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:43.448596 18219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:43.448863 18219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:43.448987 18219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:43.449189 18219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:43.449481 18219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:43.449512 18219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.449539 18219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:43.449556 18219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:43.454869 18219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46173
I20260928 23:20:43.454905 18346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46173 every 8 connection(s)
I20260928 23:20:43.455211 18219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:20:43.455610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 18219
I20260928 23:20:43.455687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:20:43.455878 392 external_mini_cluster.cc:1428] Setting key 0b59f3051c14b45ffe2dcead6ab34c08
I20260928 23:20:43.459901 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.455886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48607 (local address 127.0.98.62:41573)
0928 23:20:43.455949 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.455951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.456153 (+ 202us) server_negotiation.cc:407] Connection header received
0928 23:20:43.456770 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.456771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.456782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.456833 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:43.457457 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.457948 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.458835 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.459006 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.459055 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.459712 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:43.459713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:43.459713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:43.459717 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:43.459731 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:43.459735 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:43.459773 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.459820 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.459845 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:20:43.460076 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.455870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41573 (local address 127.0.98.3:48607)
0928 23:20:43.456039 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:43.456077 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:43.456664 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:43.456905 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:43.456909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:43.456916 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:43.457388 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.457392 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.458016 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.458016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.458775 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.458777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.459059 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.459060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.459162 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.459594 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:20:43.459595 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:43.459620 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:20:43.459642 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:43.459773 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:43.459776 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:43.459779 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:43.459821 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:20:43.459893 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260928 23:20:43.460454 18347 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41573
I20260928 23:20:43.460529 18347 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:43.460659 18347 heartbeater.cc:511] Master 127.0.98.62:41573 requested a full tablet report, sending...
I20260928 23:20:43.460944 17897 ts_manager.cc:194] Registered new tserver with Master: 45c932feecc44965ba5d450fc0f7ef29 (127.0.98.3:46173)
I20260928 23:20:43.461267 17897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48607
I20260928 23:20:43.467466 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:20:43.471465 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.468751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.0.98.62:41573)
0928 23:20:43.468817 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.468819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.468896 (+ 77us) server_negotiation.cc:407] Connection header received
0928 23:20:43.468963 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.468964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.468975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.469026 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:43.469163 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.469652 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.470389 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.470588 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.470640 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.471242 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:43.471243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:43.471243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:43.471247 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:43.471261 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:43.471265 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:43.471308 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.471361 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.471389 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:20:43.472883 17897 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:20:43.474318 17897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:20:43.480327 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.476420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34431 (local address 127.0.0.1:50552)
0928 23:20:43.476578 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:43.476616 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:43.476678 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:43.477181 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:43.477182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:43.477183 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:43.477212 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.477213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.478429 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.478430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.479190 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.479192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.480047 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.480048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.480127 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.480141 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:20:43.480202 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 23:20:43.480253 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":4,"thread_start_us":92,"threads_started":1}
I20260928 23:20:43.480451 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.476506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.0.98.2:36647)
0928 23:20:43.476912 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.476916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.476926 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:20:43.476961 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.476963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.476976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.477041 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:43.477146 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.478040 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.479609 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.480131 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.480216 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.480304 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.480338 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.480375 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":66,"threads_started":1}
I20260928 23:20:43.480599 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.476422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36647 (local address 127.0.0.1:46398)
0928 23:20:43.476773 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:43.476780 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:43.476811 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:43.477049 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:43.477049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:43.477051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:43.477093 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.477094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.478108 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.478109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.479527 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.479528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.480220 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.480221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.480278 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.480283 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:20:43.480317 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 23:20:43.480356 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":52,"threads_started":1}
I20260928 23:20:43.480634 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.476636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.0.98.1:34431)
0928 23:20:43.477052 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.477056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.477065 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:20:43.477100 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.477102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.477116 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.477187 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:43.477311 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.478372 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.479273 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.479957 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.480029 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.480117 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.480515 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.480557 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":78,"threads_started":1}
I20260928 23:20:43.480746 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.476661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.0.98.3:46173)
0928 23:20:43.476812 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.476816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.476826 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:20:43.476856 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.476858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.476869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.476941 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:43.477106 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.478303 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.479509 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.480206 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.480264 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.480340 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.480634 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.480672 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260928 23:20:43.480866 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.476422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46173 (local address 127.0.0.1:60538)
0928 23:20:43.476648 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:43.476657 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:43.476717 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:43.476964 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:43.476968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:43.476970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:43.477030 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.477032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.478388 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.478389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.479434 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.479436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.480453 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.480454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.480560 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.480564 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:20:43.480617 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:20:43.480755 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":22,"thread_start_us":91,"threads_started":1}
I20260928 23:20:43.481268 18019 tablet_service.cc:1467] Processing CreateTablet for tablet fdcfbe262ff04635ac36afebb2e7f075 (DEFAULT_TABLE table=test-table [id=81ebaaaa35704c8d9bb9778ae9b2754a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:43.481312 18150 tablet_service.cc:1467] Processing CreateTablet for tablet fdcfbe262ff04635ac36afebb2e7f075 (DEFAULT_TABLE table=test-table [id=81ebaaaa35704c8d9bb9778ae9b2754a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:43.481631 18019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcfbe262ff04635ac36afebb2e7f075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:43.481642 18150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcfbe262ff04635ac36afebb2e7f075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:43.481583 18281 tablet_service.cc:1467] Processing CreateTablet for tablet fdcfbe262ff04635ac36afebb2e7f075 (DEFAULT_TABLE table=test-table [id=81ebaaaa35704c8d9bb9778ae9b2754a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:43.481869 18281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcfbe262ff04635ac36afebb2e7f075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:43.484107 18361 tablet_bootstrap.cc:492] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8: Bootstrap starting.
I20260928 23:20:43.484746 18362 tablet_bootstrap.cc:492] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a: Bootstrap starting.
I20260928 23:20:43.484781 18361 tablet_bootstrap.cc:654] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:43.484989 18363 tablet_bootstrap.cc:492] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: Bootstrap starting.
I20260928 23:20:43.485127 18361 log.cc:824] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:43.485546 18363 tablet_bootstrap.cc:654] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:43.485852 18361 tablet_bootstrap.cc:492] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8: No bootstrap required, opened a new log
I20260928 23:20:43.485845 18363 log.cc:824] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:43.485915 18361 ts_tablet_manager.cc:1397] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:20:43.486584 18362 tablet_bootstrap.cc:654] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:43.486903 18362 log.cc:824] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:43.487399 18363 tablet_bootstrap.cc:492] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: No bootstrap required, opened a new log
I20260928 23:20:43.487463 18363 ts_tablet_manager.cc:1397] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:20:43.488481 18361 raft_consensus.cc:348] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36647 } } peers { permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } }
I20260928 23:20:43.488644 18361 raft_consensus.cc:374] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:43.488686 18361 raft_consensus.cc:729] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfc93e29e4447d0b95d0159550fc8d8, State: Initialized, Role: FOLLOWER
I20260928 23:20:43.488801 18361 consensus_queue.cc:260] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36647 } } peers { permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } }
I20260928 23:20:43.489053 18361 ts_tablet_manager.cc:1428] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:20:43.489084 18216 heartbeater.cc:503] Master 127.0.98.62:41573 was elected leader, sending a full tablet report...
I20260928 23:20:43.489815 18363 raft_consensus.cc:348] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36647 } } peers { permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } }
I20260928 23:20:43.489966 18363 raft_consensus.cc:374] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:43.490010 18363 raft_consensus.cc:729] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c932feecc44965ba5d450fc0f7ef29, State: Initialized, Role: FOLLOWER
I20260928 23:20:43.490391 18362 tablet_bootstrap.cc:492] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a: No bootstrap required, opened a new log
I20260928 23:20:43.490443 18362 ts_tablet_manager.cc:1397] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 23:20:43.491700 18363 consensus_queue.cc:260] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36647 } } peers { permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } }
I20260928 23:20:43.492004 18363 ts_tablet_manager.cc:1428] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260928 23:20:43.492013 18347 heartbeater.cc:503] Master 127.0.98.62:41573 was elected leader, sending a full tablet report...
I20260928 23:20:43.492585 18362 raft_consensus.cc:348] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36647 } } peers { permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } }
I20260928 23:20:43.492751 18362 raft_consensus.cc:374] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:43.492798 18362 raft_consensus.cc:729] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c55ae24c3e24b51be84ea1970abd57a, State: Initialized, Role: FOLLOWER
I20260928 23:20:43.492909 18362 consensus_queue.cc:260] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36647 } } peers { permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } }
I20260928 23:20:43.493119 18362 ts_tablet_manager.cc:1428] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:20:43.493185 18085 heartbeater.cc:503] Master 127.0.98.62:41573 was elected leader, sending a full tablet report...
W20260928 23:20:43.587352 18217 tablet.cc:2369] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:20:43.677953 18367 raft_consensus.cc:482] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:43.678073 18367 raft_consensus.cc:504] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36647 } } peers { permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } }
I20260928 23:20:43.678476 18367 leader_election.cc:288] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c55ae24c3e24b51be84ea1970abd57a (127.0.98.1:34431), 45c932feecc44965ba5d450fc0f7ef29 (127.0.98.3:46173)
I20260928 23:20:43.682260 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.678726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54565 (local address 127.0.98.1:34431)
0928 23:20:43.678771 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.678774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.678902 (+ 128us) server_negotiation.cc:407] Connection header received
0928 23:20:43.678953 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.678954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.678965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.679019 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:43.679168 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.679936 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.681380 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.681926 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.681992 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.682003 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.682153 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.682190 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:20:43.682260 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.678734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51143 (local address 127.0.98.3:46173)
0928 23:20:43.678777 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.678779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.678882 (+ 103us) server_negotiation.cc:407] Connection header received
0928 23:20:43.678955 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.678956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.678965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.679016 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:43.679163 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.679935 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.681380 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.681923 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.681992 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.682003 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.682153 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.682199 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:20:43.682271 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.678658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46173 (local address 127.0.98.2:51143)
0928 23:20:43.678832 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:43.678843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:43.678897 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:43.679023 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:43.679024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:43.679025 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:43.679115 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.679117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.680003 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.680003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.681297 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.681300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.681999 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.681999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.682081 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.682090 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 23:20:43.682141 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 23:20:43.682187 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":114,"thread_start_us":125,"threads_started":1}
I20260928 23:20:43.682276 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.678642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34431 (local address 127.0.98.2:54565)
0928 23:20:43.678832 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:43.678843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:43.678897 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:43.679023 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:43.679024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:43.679025 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:43.679115 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.679117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.680003 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.680003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.681297 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:43.681300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.681999 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.681999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:43.682082 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.682090 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 23:20:43.682141 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 23:20:43.682198 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":29}
I20260928 23:20:43.682592 18301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcfbe262ff04635ac36afebb2e7f075" candidate_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c932feecc44965ba5d450fc0f7ef29" is_pre_election: true
I20260928 23:20:43.682765 18301 raft_consensus.cc:2454] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfc93e29e4447d0b95d0159550fc8d8 in term 0.
I20260928 23:20:43.682934 18039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcfbe262ff04635ac36afebb2e7f075" candidate_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c55ae24c3e24b51be84ea1970abd57a" is_pre_election: true
I20260928 23:20:43.682986 18103 leader_election.cc:302] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfc93e29e4447d0b95d0159550fc8d8, 45c932feecc44965ba5d450fc0f7ef29; no voters:
I20260928 23:20:43.683068 18039 raft_consensus.cc:2454] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfc93e29e4447d0b95d0159550fc8d8 in term 0.
I20260928 23:20:43.683094 18367 raft_consensus.cc:2781] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:20:43.683131 18367 raft_consensus.cc:482] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:20:43.683156 18367 raft_consensus.cc:3037] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:43.683890 18367 raft_consensus.cc:504] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36647 } } peers { permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } }
I20260928 23:20:43.684001 18367 leader_election.cc:288] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [CANDIDATE]: Term 1 election: Requested vote from peers 0c55ae24c3e24b51be84ea1970abd57a (127.0.98.1:34431), 45c932feecc44965ba5d450fc0f7ef29 (127.0.98.3:46173)
I20260928 23:20:43.684149 18301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcfbe262ff04635ac36afebb2e7f075" candidate_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45c932feecc44965ba5d450fc0f7ef29"
I20260928 23:20:43.684155 18039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcfbe262ff04635ac36afebb2e7f075" candidate_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c55ae24c3e24b51be84ea1970abd57a"
I20260928 23:20:43.684221 18301 raft_consensus.cc:3037] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:43.684226 18039 raft_consensus.cc:3037] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:43.684913 18301 raft_consensus.cc:2454] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfc93e29e4447d0b95d0159550fc8d8 in term 1.
I20260928 23:20:43.684966 18039 raft_consensus.cc:2454] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfc93e29e4447d0b95d0159550fc8d8 in term 1.
I20260928 23:20:43.685101 18103 leader_election.cc:302] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfc93e29e4447d0b95d0159550fc8d8, 45c932feecc44965ba5d450fc0f7ef29; no voters:
I20260928 23:20:43.685207 18367 raft_consensus.cc:2781] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:43.685333 18367 raft_consensus.cc:686] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [term 1 LEADER]: Becoming Leader. State: Replica: 0dfc93e29e4447d0b95d0159550fc8d8, State: Running, Role: LEADER
I20260928 23:20:43.685418 18367 consensus_queue.cc:237] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } } peers { permanent_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36647 } } peers { permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } }
I20260928 23:20:43.686203 17897 catalog_manager.cc:5495] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dfc93e29e4447d0b95d0159550fc8d8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfc93e29e4447d0b95d0159550fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 } health_report { overall_health: UNKNOWN } } }
W20260928 23:20:43.705688 18348 tablet.cc:2369] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:20:43.707279 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.705010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.0.98.2:36647)
0928 23:20:43.705066 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:43.705068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:43.705075 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:20:43.705127 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:43.705128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:43.705143 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:43.705210 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:20:43.705358 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.705851 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.706584 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:43.706773 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:43.706825 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:43.706971 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:20:43.707067 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:20:43.707135 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:43.707173 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:20:43.707192 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:20:43.708978 18039 raft_consensus.cc:1261] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a [term 1 FOLLOWER]: Refusing update from remote peer 0dfc93e29e4447d0b95d0159550fc8d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:43.708992 18301 raft_consensus.cc:1261] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29 [term 1 FOLLOWER]: Refusing update from remote peer 0dfc93e29e4447d0b95d0159550fc8d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:43.709295 18367 consensus_queue.cc:1035] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c55ae24c3e24b51be84ea1970abd57a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:43.709426 18371 consensus_queue.cc:1035] T fdcfbe262ff04635ac36afebb2e7f075 P 0dfc93e29e4447d0b95d0159550fc8d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45c932feecc44965ba5d450fc0f7ef29" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:43.711400 18374 mvcc.cc:205] Tried to move back new op lower bound from 7334451788629483520 to 7334451788536061952. Current Snapshot: MvccSnapshot[applied={T|T < 7334451788629483520}]
I20260928 23:20:43.712175 18376 mvcc.cc:205] Tried to move back new op lower bound from 7334451788629483520 to 7334451788536061952. Current Snapshot: MvccSnapshot[applied={T|T < 7334451788629483520}]
I20260928 23:20:43.713758 17897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:43.713845 17897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:43.714533 17897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d97c8ad34f2845db82a2b5402ffefe9f: Sending DeleteTablet for 3 replicas of tablet fdcfbe262ff04635ac36afebb2e7f075
I20260928 23:20:43.714838 18150 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcfbe262ff04635ac36afebb2e7f075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:43 UTC) from {username='slave'} at 127.0.0.1:46398
I20260928 23:20:43.714942 18019 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcfbe262ff04635ac36afebb2e7f075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:43 UTC) from {username='slave'} at 127.0.0.1:50552
I20260928 23:20:43.715039 18281 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcfbe262ff04635ac36afebb2e7f075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:43 UTC) from {username='slave'} at 127.0.0.1:60538
I20260928 23:20:43.715138 18385 tablet_replica.cc:330] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a: stopping tablet replica
I20260928 23:20:43.715231 18385 raft_consensus.cc:2229] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:43.715312 18386 tablet_replica.cc:330] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: stopping tablet replica
I20260928 23:20:43.715329 18385 raft_consensus.cc:2258] T fdcfbe262ff04635ac36afebb2e7f075 P 0c55ae24c3e24b51be84ea1970abd57a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:43.715431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 17957
I20260928 23:20:43.717396 18386 raft_consensus.cc:2229] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:43.717502 18386 raft_consensus.cc:2258] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:43.717924 18386 ts_tablet_manager.cc:1905] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:43.719532 18386 ts_tablet_manager.cc:1918] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:43.719614 18386 log.cc:1196] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/fdcfbe262ff04635ac36afebb2e7f075
I20260928 23:20:43.719812 18386 ts_tablet_manager.cc:1939] T fdcfbe262ff04635ac36afebb2e7f075 P 45c932feecc44965ba5d450fc0f7ef29: Deleting consensus metadata
I20260928 23:20:43.720144 17881 catalog_manager.cc:4841] TS 45c932feecc44965ba5d450fc0f7ef29 (127.0.98.3:46173): tablet fdcfbe262ff04635ac36afebb2e7f075 (table test-table [id=81ebaaaa35704c8d9bb9778ae9b2754a]) successfully deleted
W20260928 23:20:43.720832 17882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34431 (error 108)
I20260928 23:20:43.721073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 18088
I20260928 23:20:43.721215 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.721103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34431 (local address 127.0.0.1:50554)
0928 23:20:43.721153 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:43.721188 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 23:20:43.721688 17882 catalog_manager.cc:4584] TS 0c55ae24c3e24b51be84ea1970abd57a (127.0.98.1:34431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fdcfbe262ff04635ac36afebb2e7f075: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34431: connect: Connection refused (error 111)
I20260928 23:20:43.725468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 18219
I20260928 23:20:43.725497 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:43.725398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36647 (local address 127.0.0.1:46410)
0928 23:20:43.725444 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:43.725472 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:20:43.729846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 17866
2026-09-28T23:20:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (813 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:43 dist-test-slave-djkr krb5kdc[18390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:43 dist-test-slave-djkr krb5kdc[18390](info): set up 2 sockets
Sep 28 23:20:43 dist-test-slave-djkr krb5kdc[18390](info): commencing operation
krb5kdc: starting...
W20260928 23:20:44.675057 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:44 dist-test-slave-djkr krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:20:44.784381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43183
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:44.861023 18403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:44.861222 18403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:44.861240 18403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:44.862289 18403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:44.862320 18403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:44.862332 18403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:44.862344 18403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:44.862354 18403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:44.863736 18403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:44.863916 18403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:44.864081 18403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:44.868482 18408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:44.868506 18403 server_base.cc:1034] running on GCE node
W20260928 23:20:44.868507 18411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:44.868481 18409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:44.868944 18403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:44.869226 18403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:44.870393 18403 hybrid_clock.cc:648] HybridClock initialized: now 1790637644870372 us; error 30 us; skew 500 ppm
Sep 28 23:20:44 dist-test-slave-djkr krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:44.877256 18403 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:20:44.877660 18403 webserver.cc:460] Webserver started at http://127.0.98.62:39329/ using document root <none> and password file <none>
I20260928 23:20:44.877854 18403 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:44.877897 18403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:44.877995 18403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:44.878944 18403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "ba138ba164de418e950c3cf3b4c609aa"
format_stamp: "Formatted at 2026-09-28 23:20:44 on dist-test-slave-djkr"
server_key: "202f8cda406c560d62822a9a649091fc"
server_key_iv: "8bc11b1dc49274063133973ed16fc0e7"
server_key_version: "encryptionkey@0"
I20260928 23:20:44.879225 18403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba138ba164de418e950c3cf3b4c609aa"
format_stamp: "Formatted at 2026-09-28 23:20:44 on dist-test-slave-djkr"
server_key: "202f8cda406c560d62822a9a649091fc"
server_key_iv: "8bc11b1dc49274063133973ed16fc0e7"
server_key_version: "encryptionkey@0"
I20260928 23:20:44.880391 18403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:20:44.881050 18418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:44.881201 18403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:20:44.881271 18403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "ba138ba164de418e950c3cf3b4c609aa"
format_stamp: "Formatted at 2026-09-28 23:20:44 on dist-test-slave-djkr"
server_key: "202f8cda406c560d62822a9a649091fc"
server_key_iv: "8bc11b1dc49274063133973ed16fc0e7"
server_key_version: "encryptionkey@0"
I20260928 23:20:44.881331 18403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:44.900283 18403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:44.901947 18403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:44.902064 18403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:44.905524 18403 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43183
I20260928 23:20:44.905555 18470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43183 every 8 connection(s)
I20260928 23:20:44.905864 18403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:44.906605 18471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:44.908259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 18403
I20260928 23:20:44.908370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:44.908538 392 external_mini_cluster.cc:1428] Setting key 0a05a6f06a467c2748a800b04ebabbd6
I20260928 23:20:44.908967 18471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa: Bootstrap starting.
I20260928 23:20:44.909626 18471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:44.909920 18471 log.cc:824] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:44.910526 18471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa: No bootstrap required, opened a new log
I20260928 23:20:44.912714 18471 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba138ba164de418e950c3cf3b4c609aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } }
I20260928 23:20:44.912873 18471 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:44.912902 18471 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba138ba164de418e950c3cf3b4c609aa, State: Initialized, Role: FOLLOWER
I20260928 23:20:44.913019 18471 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba138ba164de418e950c3cf3b4c609aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } }
I20260928 23:20:44.913087 18471 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:44.913123 18471 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:44.913168 18471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:44.913790 18471 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba138ba164de418e950c3cf3b4c609aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } }
I20260928 23:20:44.913902 18471 leader_election.cc:302] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba138ba164de418e950c3cf3b4c609aa; no voters:
I20260928 23:20:44.914073 18471 leader_election.cc:288] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:44.914108 18476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:44.914259 18476 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [term 1 LEADER]: Becoming Leader. State: Replica: ba138ba164de418e950c3cf3b4c609aa, State: Running, Role: LEADER
I20260928 23:20:44.914345 18476 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba138ba164de418e950c3cf3b4c609aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } }
I20260928 23:20:44.914332 18471 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:44.914799 18477 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba138ba164de418e950c3cf3b4c609aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba138ba164de418e950c3cf3b4c609aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } }
I20260928 23:20:44.914897 18477 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:44.914799 18478 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba138ba164de418e950c3cf3b4c609aa. Latest consensus state: current_term: 1 leader_uuid: "ba138ba164de418e950c3cf3b4c609aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba138ba164de418e950c3cf3b4c609aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } }
I20260928 23:20:44.915020 18478 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa [sys.catalog]: This master's current role is: LEADER
Sep 28 23:20:44 dist-test-slave-djkr krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:44.915598 18480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:44.916055 18480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:44.917573 18480 catalog_manager.cc:1347] Generated new cluster ID: 430919486f174d0ea57de21f2f25346c
I20260928 23:20:44.917621 18480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:44.919378 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:44.908973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.0.98.62:43183)
0928 23:20:44.909142 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:44.909151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:44.909179 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:20:44.909715 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:44.909725 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:44.909792 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:44.909945 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:44.910504 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:44.911864 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:44.912582 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:44.913185 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:44.913289 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:44.916027 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:44.916031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:44.916032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:44.916043 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:44.917886 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:44.918249 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:44.918252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:44.918252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:44.918352 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:44.918520 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:44.918521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:44.918521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:44.918805 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:44.918946 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:44.919028 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 23:20:44.919176 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260928 23:20:44.932503 18480 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:44.932950 18480 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:44.941260 18480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa: Generated new TSK 0
I20260928 23:20:44.941407 18480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:20:44.974076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43183
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:45.071290 18497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:45.071473 18497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:45.071494 18497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:45.072623 18497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:45.072664 18497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:45.072688 18497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:20:45.074059 18497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:45.074236 18497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:45.074402 18497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:45.078923 18505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:45.078948 18502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:45.078933 18503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:45.079146 18497 server_base.cc:1034] running on GCE node
I20260928 23:20:45.079363 18497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:45.079632 18497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:45.080767 18497 hybrid_clock.cc:648] HybridClock initialized: now 1790637645080765 us; error 39 us; skew 500 ppm
Sep 28 23:20:45 dist-test-slave-djkr krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:45.083071 18497 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:20:45.083474 18497 webserver.cc:460] Webserver started at http://127.0.98.1:33131/ using document root <none> and password file <none>
I20260928 23:20:45.083729 18497 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:45.083772 18497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:45.083865 18497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:45.084897 18497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "8442d3392a134012a8bbc130d9e00594"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "7018ae6456ed32007c44b0d30b12c4c5"
server_key_iv: "40047805ddf9f441c2593a298d0f62f5"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.085184 18497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8442d3392a134012a8bbc130d9e00594"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "7018ae6456ed32007c44b0d30b12c4c5"
server_key_iv: "40047805ddf9f441c2593a298d0f62f5"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.086344 18497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:20:45.086957 18512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.087095 18497 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:20:45.087164 18497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "8442d3392a134012a8bbc130d9e00594"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "7018ae6456ed32007c44b0d30b12c4c5"
server_key_iv: "40047805ddf9f441c2593a298d0f62f5"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.087241 18497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:45.104959 18497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:45.106812 18497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:45.106925 18497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:45.107138 18497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:45.107425 18497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:45.107456 18497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.107477 18497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:45.107501 18497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.112723 18497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42573
I20260928 23:20:45.112788 18625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42573 every 8 connection(s)
I20260928 23:20:45.113044 18497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:20:45.118718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 18497
I20260928 23:20:45.118820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:20:45.119032 392 external_mini_cluster.cc:1428] Setting key 5a32844e7cc7182a566e9af92138eeef
Sep 28 23:20:45 dist-test-slave-djkr krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:45.122038 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.113734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39387 (local address 127.0.98.62:43183)
0928 23:20:45.113775 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.113777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.113950 (+ 173us) server_negotiation.cc:407] Connection header received
0928 23:20:45.114925 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.114926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.114937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.114991 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:45.115709 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.116260 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.117223 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.117433 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.117487 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.119578 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:45.119580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:45.119580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:45.119587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:45.120817 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:45.121157 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:45.121158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:45.121158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:45.121227 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:45.121434 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:45.121435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:45.121435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:45.121546 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:45.121600 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.121850 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.121947 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:20:45.122160 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.113694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43183 (local address 127.0.98.1:39387)
0928 23:20:45.113843 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.113883 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:45.114838 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:45.115079 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:45.115083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:45.115367 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:45.115625 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.115632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.116337 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.116338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.117145 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.117147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.117498 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.117499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.117629 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.118044 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:45.118052 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:45.119483 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:45.120880 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:45.120887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:45.120889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:45.121094 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:45.121278 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:45.121279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:45.121280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:45.121381 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:45.121596 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:45.121601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:45.121737 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 23:20:45.121820 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:20:45.121960 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260928 23:20:45.122558 18626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43183
I20260928 23:20:45.122659 18626 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:45.122838 18626 heartbeater.cc:511] Master 127.0.98.62:43183 requested a full tablet report, sending...
I20260928 23:20:45.123329 18435 ts_manager.cc:194] Registered new tserver with Master: 8442d3392a134012a8bbc130d9e00594 (127.0.98.1:42573)
I20260928 23:20:45.123962 18435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39387
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:20:45.142292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43183
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:45.220515 18631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:45.220719 18631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:45.220741 18631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:45.221873 18631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:45.221912 18631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:45.221938 18631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:20:45.223368 18631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:45.223588 18631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:45.223799 18631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:45.228564 18639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:45.228657 18637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:45.228596 18636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:45.228809 18631 server_base.cc:1034] running on GCE node
I20260928 23:20:45.229029 18631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:45.229261 18631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:45.230400 18631 hybrid_clock.cc:648] HybridClock initialized: now 1790637645230383 us; error 31 us; skew 500 ppm
Sep 28 23:20:45 dist-test-slave-djkr krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:45.232856 18631 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:20:45.233278 18631 webserver.cc:460] Webserver started at http://127.0.98.2:32877/ using document root <none> and password file <none>
I20260928 23:20:45.233496 18631 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:45.233556 18631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:45.233688 18631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:45.234694 18631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "032c2975239c4d828e3ac8c1bcb2e801"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "6f63b859cb9c50c09f040a2c6478c202"
server_key_iv: "3c27ad1ff22e4ba9fb47c4cd4f60e4ad"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.235025 18631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "032c2975239c4d828e3ac8c1bcb2e801"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "6f63b859cb9c50c09f040a2c6478c202"
server_key_iv: "3c27ad1ff22e4ba9fb47c4cd4f60e4ad"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.236358 18631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:20:45.237048 18646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.237213 18631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:45.237290 18631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "032c2975239c4d828e3ac8c1bcb2e801"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "6f63b859cb9c50c09f040a2c6478c202"
server_key_iv: "3c27ad1ff22e4ba9fb47c4cd4f60e4ad"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.237351 18631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:45.244153 18631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:45.245841 18631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:45.245963 18631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:45.246143 18631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:45.246424 18631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:45.246456 18631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.246482 18631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:45.246505 18631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.251791 18631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46311
I20260928 23:20:45.251838 18759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46311 every 8 connection(s)
I20260928 23:20:45.252118 18631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:20:45.256569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 18631
I20260928 23:20:45.256666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:20:45.256855 392 external_mini_cluster.cc:1428] Setting key 45499273e1b67aeab52e20064e52e828
Sep 28 23:20:45 dist-test-slave-djkr krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:45.261618 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.252744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40575 (local address 127.0.98.62:43183)
0928 23:20:45.252808 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.252810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.252961 (+ 151us) server_negotiation.cc:407] Connection header received
0928 23:20:45.253849 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.253850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.253860 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.253917 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:45.254513 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.255025 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.256140 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.256347 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.256458 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.258727 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:45.258729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:45.258729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:45.258737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:45.260285 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:45.260685 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:45.260687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:45.260687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:45.260746 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:45.261005 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:45.261006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:45.261007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:45.261132 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:45.261191 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.261441 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.261529 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:20:45.261813 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.252699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43183 (local address 127.0.98.2:40575)
0928 23:20:45.252849 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.252887 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:45.253734 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:45.253980 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:45.253983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:45.254228 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:45.254411 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.254416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.255128 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.255129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.256038 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.256040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.256460 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.256461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.256566 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.257124 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:45.257134 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:45.258622 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:45.260358 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:45.260365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:45.260366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:45.260618 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:45.260805 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:45.260805 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:45.260806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:45.260909 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:45.261190 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:45.261198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:45.261321 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 23:20:45.261408 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 23:20:45.261606 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260928 23:20:45.262434 18760 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43183
I20260928 23:20:45.262557 18760 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:45.262735 18760 heartbeater.cc:511] Master 127.0.98.62:43183 requested a full tablet report, sending...
I20260928 23:20:45.263103 18435 ts_manager.cc:194] Registered new tserver with Master: 032c2975239c4d828e3ac8c1bcb2e801 (127.0.98.2:46311)
I20260928 23:20:45.263451 18435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40575
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:20:45.281737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43183
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:45.359905 18765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:45.360106 18765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:45.360127 18765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:45.361275 18765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:45.361315 18765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:45.361340 18765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:20:45.362803 18765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:45.363015 18765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:45.363224 18765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:45.367887 18770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:45.367949 18773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:45.368065 18765 server_base.cc:1034] running on GCE node
W20260928 23:20:45.367893 18771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:45.368302 18765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:45.368515 18765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:45.369656 18765 hybrid_clock.cc:648] HybridClock initialized: now 1790637645369640 us; error 33 us; skew 500 ppm
Sep 28 23:20:45 dist-test-slave-djkr krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:45.372138 18765 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:20:45.372551 18765 webserver.cc:460] Webserver started at http://127.0.98.3:33295/ using document root <none> and password file <none>
I20260928 23:20:45.372752 18765 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:45.372803 18765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:45.372922 18765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:45.373915 18765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "389ea722d3464f91b4701e572081879c"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "5f331415b66b85a81872764ba5ba5cca"
server_key_iv: "439da7fe8b5b40c00bb0bed3d9316899"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.374243 18765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "389ea722d3464f91b4701e572081879c"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "5f331415b66b85a81872764ba5ba5cca"
server_key_iv: "439da7fe8b5b40c00bb0bed3d9316899"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.375434 18765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:20:45.376153 18780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.376353 18765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.376441 18765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "389ea722d3464f91b4701e572081879c"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "5f331415b66b85a81872764ba5ba5cca"
server_key_iv: "439da7fe8b5b40c00bb0bed3d9316899"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.376508 18765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:45.390758 18765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:45.392705 18765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:45.392822 18765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:45.393021 18765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:45.393301 18765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:45.393337 18765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.393357 18765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:45.393379 18765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.398406 18765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45689
I20260928 23:20:45.398470 18893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45689 every 8 connection(s)
I20260928 23:20:45.398731 18765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:20:45 dist-test-slave-djkr krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:45.405997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 18765
I20260928 23:20:45.406088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:20:45.406251 392 external_mini_cluster.cc:1428] Setting key 75193e3f9c41af8232585c618f9076e0
I20260928 23:20:45.407445 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.399361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60131 (local address 127.0.98.62:43183)
0928 23:20:45.399413 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.399415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.399632 (+ 217us) server_negotiation.cc:407] Connection header received
0928 23:20:45.400528 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.400529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.400538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.400576 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:45.401129 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.401611 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.402635 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.402834 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.402911 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.404838 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:45.404840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:45.404840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:45.404847 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:45.406101 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:45.406499 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:45.406500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:45.406501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:45.406561 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:45.406835 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:45.406836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:45.406837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:45.406968 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:45.407031 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.407311 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.407380 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:20:45.407639 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.399341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43183 (local address 127.0.98.3:60131)
0928 23:20:45.399494 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.399530 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:45.400433 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:45.400644 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:45.400647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:45.400883 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:45.401068 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.401072 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.401679 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.401680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.402538 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.402540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.402921 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.402922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.403040 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.403462 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:45.403469 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:45.404758 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:45.406184 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:45.406191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:45.406193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:45.406436 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:45.406624 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:45.406625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:45.406626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:45.406772 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:45.407025 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:45.407031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:45.407190 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 23:20:45.407285 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 23:20:45.407419 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260928 23:20:45.408067 18894 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43183
I20260928 23:20:45.408154 18894 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:45.408324 18894 heartbeater.cc:511] Master 127.0.98.62:43183 requested a full tablet report, sending...
I20260928 23:20:45.408646 18435 ts_manager.cc:194] Registered new tserver with Master: 389ea722d3464f91b4701e572081879c (127.0.98.3:45689)
I20260928 23:20:45.409000 18435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60131
I20260928 23:20:45.417878 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:20:45.425034 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.419195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.0.98.62:43183)
0928 23:20:45.419243 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.419245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.419328 (+ 83us) server_negotiation.cc:407] Connection header received
0928 23:20:45.419373 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.419373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.419383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.419441 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:45.419997 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.420481 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.421174 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.421349 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.421390 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.422604 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:45.422605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:45.422605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:45.422611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:45.423872 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:45.424259 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:45.424259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:45.424260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:45.424308 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:45.424539 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:45.424539 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:45.424540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:45.424664 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:45.424770 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.424911 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.424964 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:20:45.426736 18435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:20:45.428426 18435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:20:45.434665 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.430609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.0.98.1:42573)
0928 23:20:45.430798 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.430804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.431097 (+ 293us) server_negotiation.cc:407] Connection header received
0928 23:20:45.431142 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.431145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.431159 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.431224 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:45.431447 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.432208 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.433557 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.434267 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.434336 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.434397 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.434537 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.434582 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260928 23:20:45.434653 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.430483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42573 (local address 127.0.0.1:35476)
0928 23:20:45.431014 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.431025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:45.431085 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:45.431312 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:45.431313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:45.431315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:45.431375 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.431377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.432300 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.432301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.433467 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.433469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.434351 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.434352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.434437 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.434451 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:20:45.434506 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 23:20:45.434552 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260928 23:20:45.434715 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.430483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46311 (local address 127.0.0.1:59754)
0928 23:20:45.430799 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.430839 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:45.430894 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:45.431149 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:45.431152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:45.431155 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:45.431266 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.431268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.432619 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.432620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.433440 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.433443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.434432 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.434433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.434488 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.434502 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:20:45.434554 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:20:45.434593 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":63,"threads_started":1}
I20260928 23:20:45.434700 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.430804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.0.98.2:46311)
0928 23:20:45.430996 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.431000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.431009 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:20:45.431043 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.431045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.431056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.431128 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:45.431346 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.432508 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.433536 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.434267 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.434335 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.434397 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.434583 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.434629 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260928 23:20:45.435089 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.430688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45689 (local address 127.0.0.1:47462)
0928 23:20:45.431151 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.431159 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:45.431197 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:45.431463 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:45.431464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:45.431466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:45.431512 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.431513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.432728 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.432729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.433747 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.433749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.434865 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.434866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.434939 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.434943 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:20:45.434989 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 23:20:45.435033 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":56,"threads_started":1}
I20260928 23:20:45.435127 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.430820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.0.98.3:45689)
0928 23:20:45.431014 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.431018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.431250 (+ 232us) server_negotiation.cc:407] Connection header received
0928 23:20:45.431293 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.431296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.431311 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.431380 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:45.431608 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.432635 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.433815 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.434275 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.434342 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.434399 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.435008 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.435055 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
I20260928 23:20:45.435705 18694 tablet_service.cc:1467] Processing CreateTablet for tablet 77e3febf2bd44e0a9e5a74af0c4d5736 (DEFAULT_TABLE table=test-table [id=649231fb27924fad8e7f89231b265d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:45.435796 18828 tablet_service.cc:1467] Processing CreateTablet for tablet 77e3febf2bd44e0a9e5a74af0c4d5736 (DEFAULT_TABLE table=test-table [id=649231fb27924fad8e7f89231b265d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:45.435796 18560 tablet_service.cc:1467] Processing CreateTablet for tablet 77e3febf2bd44e0a9e5a74af0c4d5736 (DEFAULT_TABLE table=test-table [id=649231fb27924fad8e7f89231b265d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:45.436072 18694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e3febf2bd44e0a9e5a74af0c4d5736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:45.436116 18560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e3febf2bd44e0a9e5a74af0c4d5736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:45.436115 18828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e3febf2bd44e0a9e5a74af0c4d5736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:45.438488 18908 tablet_bootstrap.cc:492] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801: Bootstrap starting.
I20260928 23:20:45.439011 18908 tablet_bootstrap.cc:654] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:45.439568 18908 log.cc:824] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:45.439584 18909 tablet_bootstrap.cc:492] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: Bootstrap starting.
I20260928 23:20:45.439832 18910 tablet_bootstrap.cc:492] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594: Bootstrap starting.
I20260928 23:20:45.440214 18909 tablet_bootstrap.cc:654] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:45.440384 18908 tablet_bootstrap.cc:492] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801: No bootstrap required, opened a new log
I20260928 23:20:45.440443 18910 tablet_bootstrap.cc:654] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:45.440455 18908 ts_tablet_manager.cc:1397] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:20:45.440729 18909 log.cc:824] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:45.440966 18910 log.cc:824] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:45.441406 18909 tablet_bootstrap.cc:492] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: No bootstrap required, opened a new log
I20260928 23:20:45.441473 18909 ts_tablet_manager.cc:1397] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:20:45.441705 18910 tablet_bootstrap.cc:492] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594: No bootstrap required, opened a new log
I20260928 23:20:45.441772 18910 ts_tablet_manager.cc:1397] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:20:45.442410 18908 raft_consensus.cc:348] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032c2975239c4d828e3ac8c1bcb2e801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46311 } } peers { permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } } peers { permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 } }
I20260928 23:20:45.442603 18908 raft_consensus.cc:374] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:45.442647 18908 raft_consensus.cc:729] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032c2975239c4d828e3ac8c1bcb2e801, State: Initialized, Role: FOLLOWER
I20260928 23:20:45.442776 18908 consensus_queue.cc:260] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032c2975239c4d828e3ac8c1bcb2e801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46311 } } peers { permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } } peers { permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 } }
I20260928 23:20:45.443061 18760 heartbeater.cc:503] Master 127.0.98.62:43183 was elected leader, sending a full tablet report...
I20260928 23:20:45.443061 18908 ts_tablet_manager.cc:1428] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:20:45.443450 18909 raft_consensus.cc:348] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032c2975239c4d828e3ac8c1bcb2e801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46311 } } peers { permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } } peers { permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 } }
I20260928 23:20:45.443642 18909 raft_consensus.cc:374] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:45.443684 18909 raft_consensus.cc:729] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389ea722d3464f91b4701e572081879c, State: Initialized, Role: FOLLOWER
I20260928 23:20:45.443653 18910 raft_consensus.cc:348] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032c2975239c4d828e3ac8c1bcb2e801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46311 } } peers { permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } } peers { permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 } }
I20260928 23:20:45.443828 18910 raft_consensus.cc:374] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:45.443801 18909 consensus_queue.cc:260] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032c2975239c4d828e3ac8c1bcb2e801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46311 } } peers { permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } } peers { permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 } }
I20260928 23:20:45.443871 18910 raft_consensus.cc:729] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8442d3392a134012a8bbc130d9e00594, State: Initialized, Role: FOLLOWER
I20260928 23:20:45.443988 18910 consensus_queue.cc:260] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032c2975239c4d828e3ac8c1bcb2e801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46311 } } peers { permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } } peers { permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 } }
I20260928 23:20:45.444208 18909 ts_tablet_manager.cc:1428] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 23:20:45.444265 18626 heartbeater.cc:503] Master 127.0.98.62:43183 was elected leader, sending a full tablet report...
I20260928 23:20:45.444310 18894 heartbeater.cc:503] Master 127.0.98.62:43183 was elected leader, sending a full tablet report...
I20260928 23:20:45.444255 18910 ts_tablet_manager.cc:1428] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260928 23:20:45.502630 18761 tablet.cc:2369] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:20:45.560976 18914 raft_consensus.cc:482] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:45.561117 18914 raft_consensus.cc:504] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032c2975239c4d828e3ac8c1bcb2e801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46311 } } peers { permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } } peers { permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 } }
I20260928 23:20:45.561498 18914 leader_election.cc:288] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8442d3392a134012a8bbc130d9e00594 (127.0.98.1:42573), 389ea722d3464f91b4701e572081879c (127.0.98.3:45689)
I20260928 23:20:45.564412 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.561773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37883 (local address 127.0.98.3:45689)
0928 23:20:45.561834 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.561836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.562096 (+ 260us) server_negotiation.cc:407] Connection header received
0928 23:20:45.562138 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.562139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.562148 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.562191 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:45.562331 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.562816 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.563722 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.564091 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.564147 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.564158 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.564302 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.564344 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:20:45.564419 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.561803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52863 (local address 127.0.98.1:42573)
0928 23:20:45.561854 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.561856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.561865 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:20:45.561893 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.561893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.561904 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.561959 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:45.562146 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.562715 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.563732 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.564097 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.564160 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.564170 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.564323 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.564358 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:20:45.564424 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.561705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42573 (local address 127.0.98.2:52863)
0928 23:20:45.561746 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.561757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:45.561808 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:45.561975 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:45.561976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:45.561978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:45.562062 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.562064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.562792 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.562793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.563644 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.563647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.564172 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.564173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.564257 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.564260 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:45.564309 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:20:45.564345 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":3}
I20260928 23:20:45.564427 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.561705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45689 (local address 127.0.98.2:37883)
0928 23:20:45.562014 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.562024 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:45.562083 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:45.562206 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:45.562206 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:45.562208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:45.562276 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.562277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.562889 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.562889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.563647 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.563649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.564163 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.564164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.564239 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.564242 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:45.564282 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 23:20:45.564318 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":150,"thread_start_us":105,"threads_started":1}
I20260928 23:20:45.564747 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e3febf2bd44e0a9e5a74af0c4d5736" candidate_uuid: "032c2975239c4d828e3ac8c1bcb2e801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8442d3392a134012a8bbc130d9e00594" is_pre_election: true
I20260928 23:20:45.564894 18580 raft_consensus.cc:2454] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032c2975239c4d828e3ac8c1bcb2e801 in term 0.
I20260928 23:20:45.565104 18650 leader_election.cc:302] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032c2975239c4d828e3ac8c1bcb2e801, 8442d3392a134012a8bbc130d9e00594; no voters:
I20260928 23:20:45.565114 18848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e3febf2bd44e0a9e5a74af0c4d5736" candidate_uuid: "032c2975239c4d828e3ac8c1bcb2e801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "389ea722d3464f91b4701e572081879c" is_pre_election: true
I20260928 23:20:45.565243 18848 raft_consensus.cc:2454] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 032c2975239c4d828e3ac8c1bcb2e801 in term 0.
I20260928 23:20:45.565237 18914 raft_consensus.cc:2781] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:20:45.565284 18914 raft_consensus.cc:482] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:20:45.565305 18914 raft_consensus.cc:3037] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:45.565953 18914 raft_consensus.cc:504] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032c2975239c4d828e3ac8c1bcb2e801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46311 } } peers { permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } } peers { permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 } }
I20260928 23:20:45.566069 18914 leader_election.cc:288] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [CANDIDATE]: Term 1 election: Requested vote from peers 8442d3392a134012a8bbc130d9e00594 (127.0.98.1:42573), 389ea722d3464f91b4701e572081879c (127.0.98.3:45689)
I20260928 23:20:45.566238 18848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e3febf2bd44e0a9e5a74af0c4d5736" candidate_uuid: "032c2975239c4d828e3ac8c1bcb2e801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "389ea722d3464f91b4701e572081879c"
I20260928 23:20:45.566282 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e3febf2bd44e0a9e5a74af0c4d5736" candidate_uuid: "032c2975239c4d828e3ac8c1bcb2e801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8442d3392a134012a8bbc130d9e00594"
I20260928 23:20:45.566318 18848 raft_consensus.cc:3037] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:45.566359 18580 raft_consensus.cc:3037] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:45.567149 18848 raft_consensus.cc:2454] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032c2975239c4d828e3ac8c1bcb2e801 in term 1.
I20260928 23:20:45.567150 18580 raft_consensus.cc:2454] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 032c2975239c4d828e3ac8c1bcb2e801 in term 1.
I20260928 23:20:45.567305 18650 leader_election.cc:302] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 032c2975239c4d828e3ac8c1bcb2e801, 8442d3392a134012a8bbc130d9e00594; no voters:
I20260928 23:20:45.567385 18914 raft_consensus.cc:2781] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:45.567528 18914 raft_consensus.cc:686] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 1 LEADER]: Becoming Leader. State: Replica: 032c2975239c4d828e3ac8c1bcb2e801, State: Running, Role: LEADER
I20260928 23:20:45.567652 18914 consensus_queue.cc:237] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032c2975239c4d828e3ac8c1bcb2e801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46311 } } peers { permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } } peers { permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 } }
I20260928 23:20:45.568426 18435 catalog_manager.cc:5495] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 reported cstate change: term changed from 0 to 1, leader changed from <none> to 032c2975239c4d828e3ac8c1bcb2e801 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "032c2975239c4d828e3ac8c1bcb2e801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "032c2975239c4d828e3ac8c1bcb2e801" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 } health_report { overall_health: UNKNOWN } } }
I20260928 23:20:45.582718 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.579788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.0.98.2:46311)
0928 23:20:45.579896 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.579897 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.579906 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:20:45.579933 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.579934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.579951 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.580012 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:20:45.580160 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.580869 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.581868 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.582167 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.582235 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.582363 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:20:45.582473 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:20:45.582572 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.582600 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.582610 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 23:20:45.584640 18848 raft_consensus.cc:1261] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c [term 1 FOLLOWER]: Refusing update from remote peer 032c2975239c4d828e3ac8c1bcb2e801: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:45.584666 18580 raft_consensus.cc:1261] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594 [term 1 FOLLOWER]: Refusing update from remote peer 032c2975239c4d828e3ac8c1bcb2e801: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:45.585043 18914 consensus_queue.cc:1035] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "389ea722d3464f91b4701e572081879c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:45.585451 18914 consensus_queue.cc:1035] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8442d3392a134012a8bbc130d9e00594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:45.588936 18926 mvcc.cc:205] Tried to move back new op lower bound from 7334451796312055808 to 7334451796245585920. Current Snapshot: MvccSnapshot[applied={T|T < 7334451796312055808}]
I20260928 23:20:45.589663 18435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:45.589742 18435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:45.590417 18435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba138ba164de418e950c3cf3b4c609aa: Sending DeleteTablet for 3 replicas of tablet 77e3febf2bd44e0a9e5a74af0c4d5736
I20260928 23:20:45.590713 18560 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e3febf2bd44e0a9e5a74af0c4d5736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35476
I20260928 23:20:45.590767 18828 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e3febf2bd44e0a9e5a74af0c4d5736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47462
I20260928 23:20:45.590796 18694 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e3febf2bd44e0a9e5a74af0c4d5736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59754
I20260928 23:20:45.591073 18931 tablet_replica.cc:330] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594: stopping tablet replica
I20260928 23:20:45.591140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 18497
I20260928 23:20:45.591221 18931 raft_consensus.cc:2229] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:45.591334 18931 raft_consensus.cc:2258] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 8442d3392a134012a8bbc130d9e00594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:45.595677 18932 tablet_replica.cc:330] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: stopping tablet replica
I20260928 23:20:45.595783 18932 raft_consensus.cc:2229] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:45.595875 18932 raft_consensus.cc:2258] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:45.596015 18933 tablet_replica.cc:330] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801: stopping tablet replica
I20260928 23:20:45.596148 18933 raft_consensus.cc:2229] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:20:45.596300 18932 ts_tablet_manager.cc:1905] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:45.596313 18933 raft_consensus.cc:2258] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:45.596933 18933 ts_tablet_manager.cc:1905] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 032c2975239c4d828e3ac8c1bcb2e801: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:20:45.596999 18422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42573 (error 108)
I20260928 23:20:45.597688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 18631
I20260928 23:20:45.597723 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.597581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42573 (local address 127.0.0.1:35480)
0928 23:20:45.597630 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.597690 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:20:45.598327 18932 ts_tablet_manager.cc:1918] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:45.598398 18932 log.cc:1196] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/77e3febf2bd44e0a9e5a74af0c4d5736
W20260928 23:20:45.598517 18422 catalog_manager.cc:4584] TS 8442d3392a134012a8bbc130d9e00594 (127.0.98.1:42573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77e3febf2bd44e0a9e5a74af0c4d5736: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42573: connect: Connection refused (error 111)
I20260928 23:20:45.598662 18932 ts_tablet_manager.cc:1939] T 77e3febf2bd44e0a9e5a74af0c4d5736 P 389ea722d3464f91b4701e572081879c: Deleting consensus metadata
I20260928 23:20:45.599228 18420 catalog_manager.cc:4841] TS 389ea722d3464f91b4701e572081879c (127.0.98.3:45689): tablet 77e3febf2bd44e0a9e5a74af0c4d5736 (table test-table [id=649231fb27924fad8e7f89231b265d58]) successfully deleted
I20260928 23:20:45.601962 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.601856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46311 (local address 127.0.0.1:59782)
0928 23:20:45.601905 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.601934 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 23:20:45.602229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 18765
I20260928 23:20:45.606146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 18403
2026-09-28T23:20:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1876 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T23:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:45Z Disabled control of system clock
I20260928 23:20:45.630959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33305
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:20:45.709793 18938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:45.710003 18938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:45.710022 18938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:45.711119 18938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:45.711150 18938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:45.711161 18938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:45.711172 18938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:45.711184 18938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:45.712509 18938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:45.712709 18938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:45.712934 18938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:45.717404 18943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:45.717519 18938 server_base.cc:1034] running on GCE node
W20260928 23:20:45.717404 18946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:45.717463 18944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:45.717897 18938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:45.718175 18938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:45.719338 18938 hybrid_clock.cc:648] HybridClock initialized: now 1790637645719323 us; error 30 us; skew 500 ppm
I20260928 23:20:45.720633 18938 webserver.cc:460] Webserver started at http://127.0.98.62:35347/ using document root <none> and password file <none>
I20260928 23:20:45.720856 18938 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:45.720902 18938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:45.721028 18938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:45.722024 18938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "2c8247e581f84bc0872b1c7c3e9f3300"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "c4ba5af5b141642f93b0c0131f31de8a"
server_key_iv: "48c86f736086aa9df5a49646d48b44cd"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.722366 18938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c8247e581f84bc0872b1c7c3e9f3300"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "c4ba5af5b141642f93b0c0131f31de8a"
server_key_iv: "48c86f736086aa9df5a49646d48b44cd"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.723627 18938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:20:45.724368 18952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.724566 18938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:45.724656 18938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "2c8247e581f84bc0872b1c7c3e9f3300"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "c4ba5af5b141642f93b0c0131f31de8a"
server_key_iv: "48c86f736086aa9df5a49646d48b44cd"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.724723 18938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:45.755481 18938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:45.755831 18938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:45.755962 18938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:45.759506 18938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33305
I20260928 23:20:45.759600 19004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33305 every 8 connection(s)
I20260928 23:20:45.759897 18938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:45.760792 19005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:45.763137 19005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300: Bootstrap starting.
I20260928 23:20:45.763722 19005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:45.763979 19005 log.cc:824] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:45.764583 19005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300: No bootstrap required, opened a new log
I20260928 23:20:45.764875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 18938
I20260928 23:20:45.764966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:45.765139 392 external_mini_cluster.cc:1428] Setting key ee9070df9b6b4e05b99aea39351bf4a0
I20260928 23:20:45.766414 19005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8247e581f84bc0872b1c7c3e9f3300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33305 } }
I20260928 23:20:45.766548 19005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:45.766579 19005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8247e581f84bc0872b1c7c3e9f3300, State: Initialized, Role: FOLLOWER
I20260928 23:20:45.766724 19005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8247e581f84bc0872b1c7c3e9f3300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33305 } }
I20260928 23:20:45.766795 19005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:45.766825 19005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:45.766871 19005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:45.767367 19005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8247e581f84bc0872b1c7c3e9f3300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33305 } }
I20260928 23:20:45.767457 19005 leader_election.cc:302] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c8247e581f84bc0872b1c7c3e9f3300; no voters:
I20260928 23:20:45.767666 19005 leader_election.cc:288] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:45.767668 19010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:45.767905 19010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [term 1 LEADER]: Becoming Leader. State: Replica: 2c8247e581f84bc0872b1c7c3e9f3300, State: Running, Role: LEADER
I20260928 23:20:45.767992 19010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8247e581f84bc0872b1c7c3e9f3300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33305 } }
I20260928 23:20:45.768042 19005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:45.768368 19011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c8247e581f84bc0872b1c7c3e9f3300. Latest consensus state: current_term: 1 leader_uuid: "2c8247e581f84bc0872b1c7c3e9f3300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8247e581f84bc0872b1c7c3e9f3300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33305 } } }
I20260928 23:20:45.768443 19011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:45.768610 19012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c8247e581f84bc0872b1c7c3e9f3300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8247e581f84bc0872b1c7c3e9f3300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33305 } } }
I20260928 23:20:45.768671 19012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:45.768982 19016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:45.769438 19016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:45.771144 19016 catalog_manager.cc:1347] Generated new cluster ID: 5c30a57418a641708e36413e2c558433
I20260928 23:20:45.771186 19016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:45.771627 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.765616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.0.98.62:33305)
0928 23:20:45.765824 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.765831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.765855 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:20:45.766362 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.766369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.766427 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.766559 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:45.766779 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.768253 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.769556 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.770143 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.770259 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.770893 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:45.770897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:45.770897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:45.771103 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:45.771143 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:45.771158 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:45.771228 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.771337 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.771411 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260928 23:20:45.778457 19016 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:45.779139 19016 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:45.787519 19016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300: Generated new TSK 0
I20260928 23:20:45.787730 19016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:45.789672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33305
--builtin_ntp_servers=127.0.98.20:40109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:20:45.870678 19029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:45.870882 19029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:45.870903 19029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:45.872102 19029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:45.872140 19029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:45.872164 19029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:20:45.873461 19029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:45.873683 19029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:45.873958 19029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:45.878630 19034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:45.878623 19037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:45.878785 19029 server_base.cc:1034] running on GCE node
W20260928 23:20:45.878633 19035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:45.879112 19029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:45.879338 19029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:45.880474 19029 hybrid_clock.cc:648] HybridClock initialized: now 1790637645880461 us; error 31 us; skew 500 ppm
I20260928 23:20:45.881598 19029 webserver.cc:460] Webserver started at http://127.0.98.1:39895/ using document root <none> and password file <none>
I20260928 23:20:45.881798 19029 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:45.881847 19029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:45.881964 19029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:45.883009 19029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "285eff64a8ae45e1acd2ceb868c34750"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "6d22760f7204263c06c1b7e1d198c181"
server_key_iv: "bc310553574f90c666d270aaf365e6f8"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.883325 19029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "285eff64a8ae45e1acd2ceb868c34750"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "6d22760f7204263c06c1b7e1d198c181"
server_key_iv: "bc310553574f90c666d270aaf365e6f8"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.884675 19029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:20:45.885346 19043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.885492 19029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:20:45.885569 19029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "285eff64a8ae45e1acd2ceb868c34750"
format_stamp: "Formatted at 2026-09-28 23:20:45 on dist-test-slave-djkr"
server_key: "6d22760f7204263c06c1b7e1d198c181"
server_key_iv: "bc310553574f90c666d270aaf365e6f8"
server_key_version: "encryptionkey@0"
I20260928 23:20:45.885658 19029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:45.919847 19029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:45.920195 19029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:45.920336 19029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:45.920573 19029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:45.920923 19029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:45.920956 19029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.920986 19029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:45.921010 19029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:45.926838 19029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35021
I20260928 23:20:45.926910 19156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35021 every 8 connection(s)
I20260928 23:20:45.927277 19029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:20:45.932777 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.928051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46197 (local address 127.0.98.62:33305)
0928 23:20:45.928099 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:45.928101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:45.928330 (+ 229us) server_negotiation.cc:407] Connection header received
0928 23:20:45.928879 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:45.928880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:45.928890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:45.928940 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:45.929635 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.930132 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.931191 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.931383 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.931443 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.932506 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:45.932507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:45.932507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:45.932513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:45.932526 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:45.932530 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:45.932571 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:45.932677 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 23:20:45.932709 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:20:45.933029 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:45.928035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33305 (local address 127.0.98.1:46197)
0928 23:20:45.928217 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:45.928255 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:45.928786 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:45.929007 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:45.929011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:45.929019 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:45.929549 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.929554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.930227 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.930228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.931096 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:45.931100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:45.931498 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:45.931500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:45.931711 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:45.932347 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:20:45.932348 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:45.932397 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:20:45.932428 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:45.932572 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:45.932575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:45.932577 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:20:45.932679 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 23:20:45.932776 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260928 23:20:45.933645 19157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33305
I20260928 23:20:45.933745 19157 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:45.933914 19157 heartbeater.cc:511] Master 127.0.98.62:33305 requested a full tablet report, sending...
I20260928 23:20:45.934476 18969 ts_manager.cc:194] Registered new tserver with Master: 285eff64a8ae45e1acd2ceb868c34750 (127.0.98.1:35021)
I20260928 23:20:45.934494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 19029
I20260928 23:20:45.934602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:20:45.934805 392 external_mini_cluster.cc:1428] Setting key 47085c25582e0c162ceb9dcbfbb2ebab
I20260928 23:20:45.935425 18969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46197
I20260928 23:20:45.935981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33305
--builtin_ntp_servers=127.0.98.20:40109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:20:46.015125 19160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:46.015319 19160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:46.015339 19160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:46.016446 19160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:46.016485 19160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:46.016510 19160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:20:46.017825 19160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:46.018034 19160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:46.018309 19160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:46.023051 19168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:46.023303 19160 server_base.cc:1034] running on GCE node
W20260928 23:20:46.023058 19166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:46.023058 19165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:46.023710 19160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:46.023947 19160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:46.025082 19160 hybrid_clock.cc:648] HybridClock initialized: now 1790637646025075 us; error 28 us; skew 500 ppm
I20260928 23:20:46.026276 19160 webserver.cc:460] Webserver started at http://127.0.98.2:34861/ using document root <none> and password file <none>
I20260928 23:20:46.026491 19160 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:46.026539 19160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:46.026664 19160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:46.027844 19160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd3bf0c28af441b99a2cf4fff1cf540d"
format_stamp: "Formatted at 2026-09-28 23:20:46 on dist-test-slave-djkr"
server_key: "209d7821859cb9aecb516acadb66b4c9"
server_key_iv: "758ccae8d70ad0c1c19f01f6d8b97c90"
server_key_version: "encryptionkey@0"
I20260928 23:20:46.028198 19160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd3bf0c28af441b99a2cf4fff1cf540d"
format_stamp: "Formatted at 2026-09-28 23:20:46 on dist-test-slave-djkr"
server_key: "209d7821859cb9aecb516acadb66b4c9"
server_key_iv: "758ccae8d70ad0c1c19f01f6d8b97c90"
server_key_version: "encryptionkey@0"
I20260928 23:20:46.029388 19160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:46.030081 19174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:46.030215 19160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:46.030294 19160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "cd3bf0c28af441b99a2cf4fff1cf540d"
format_stamp: "Formatted at 2026-09-28 23:20:46 on dist-test-slave-djkr"
server_key: "209d7821859cb9aecb516acadb66b4c9"
server_key_iv: "758ccae8d70ad0c1c19f01f6d8b97c90"
server_key_version: "encryptionkey@0"
I20260928 23:20:46.030364 19160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:46.044874 19160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:46.045166 19160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:46.045296 19160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:46.045514 19160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:46.045843 19160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:46.045876 19160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:46.045907 19160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:46.045923 19160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:46.051262 19160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43135
I20260928 23:20:46.051317 19287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43135 every 8 connection(s)
I20260928 23:20:46.051642 19160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:20:46.056579 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.052347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53373 (local address 127.0.98.62:33305)
0928 23:20:46.052424 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:46.052426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:46.052599 (+ 173us) server_negotiation.cc:407] Connection header received
0928 23:20:46.053081 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:46.053082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:46.053091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:46.053140 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:46.053765 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.054248 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.055236 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.055419 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.055468 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.056222 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:46.056224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:46.056224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:46.056228 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:46.056242 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:46.056246 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:46.056288 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:46.056470 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 23:20:46.056518 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:20:46.056766 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.052320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33305 (local address 127.0.98.2:53373)
0928 23:20:46.052477 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:46.052521 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:46.053006 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:46.053196 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:46.053200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:46.053208 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:46.053690 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.053694 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.054345 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.054346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.055154 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.055156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.055490 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.055491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.055625 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.056095 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:20:46.056095 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:46.056123 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:20:46.056146 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:46.056360 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:46.056366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:46.056369 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:46.056440 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 23:20:46.056549 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260928 23:20:46.057229 19288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33305
I20260928 23:20:46.057313 19288 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:46.057489 19288 heartbeater.cc:511] Master 127.0.98.62:33305 requested a full tablet report, sending...
I20260928 23:20:46.057863 18969 ts_manager.cc:194] Registered new tserver with Master: cd3bf0c28af441b99a2cf4fff1cf540d (127.0.98.2:43135)
I20260928 23:20:46.058229 18969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53373
I20260928 23:20:46.060635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 19160
I20260928 23:20:46.060738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:20:46.060920 392 external_mini_cluster.cc:1428] Setting key 0ab7520bafb69384e17b40e0f14c9ee3
I20260928 23:20:46.061949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33305
--builtin_ntp_servers=127.0.98.20:40109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:20:46.140511 19291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:46.140710 19291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:46.140729 19291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:46.141842 19291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:46.141881 19291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:46.141904 19291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:20:46.143181 19291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:46.143399 19291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:46.143694 19291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:46.148355 19297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:46.148365 19296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:46.148443 19291 server_base.cc:1034] running on GCE node
W20260928 23:20:46.148360 19299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:46.148752 19291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:46.148967 19291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:46.150115 19291 hybrid_clock.cc:648] HybridClock initialized: now 1790637646150101 us; error 30 us; skew 500 ppm
I20260928 23:20:46.151302 19291 webserver.cc:460] Webserver started at http://127.0.98.3:37881/ using document root <none> and password file <none>
I20260928 23:20:46.151508 19291 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:46.151580 19291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:46.151734 19291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:46.152734 19291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "722e8f4991c44e6cb26c01ef0e8366b1"
format_stamp: "Formatted at 2026-09-28 23:20:46 on dist-test-slave-djkr"
server_key: "6c4281579f3f3fac43a9fd1d2f1a26b0"
server_key_iv: "c942e850012d05c85b2cdf1612a37f31"
server_key_version: "encryptionkey@0"
I20260928 23:20:46.153054 19291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "722e8f4991c44e6cb26c01ef0e8366b1"
format_stamp: "Formatted at 2026-09-28 23:20:46 on dist-test-slave-djkr"
server_key: "6c4281579f3f3fac43a9fd1d2f1a26b0"
server_key_iv: "c942e850012d05c85b2cdf1612a37f31"
server_key_version: "encryptionkey@0"
I20260928 23:20:46.154207 19291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:20:46.154819 19305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:46.154973 19291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:46.155047 19291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "722e8f4991c44e6cb26c01ef0e8366b1"
format_stamp: "Formatted at 2026-09-28 23:20:46 on dist-test-slave-djkr"
server_key: "6c4281579f3f3fac43a9fd1d2f1a26b0"
server_key_iv: "c942e850012d05c85b2cdf1612a37f31"
server_key_version: "encryptionkey@0"
I20260928 23:20:46.155110 19291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:46.198319 19291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:46.198637 19291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:46.198766 19291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:46.198995 19291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:46.199339 19291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:46.199373 19291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:46.199404 19291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:46.199429 19291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:46.205288 19291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42511
I20260928 23:20:46.205351 19418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42511 every 8 connection(s)
I20260928 23:20:46.205669 19291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:20:46.206791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 19291
I20260928 23:20:46.206888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:20:46.207096 392 external_mini_cluster.cc:1428] Setting key 4668ab7db51515866983d73705300c9a
I20260928 23:20:46.210841 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.206352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36009 (local address 127.0.98.62:33305)
0928 23:20:46.206413 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:46.206415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:46.206651 (+ 236us) server_negotiation.cc:407] Connection header received
0928 23:20:46.207297 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:46.207299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:46.207311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:46.207369 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:46.208206 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.208685 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.209614 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.209796 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.209851 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.210637 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:46.210639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:46.210639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:46.210642 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:46.210656 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:46.210660 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:46.210708 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:46.210765 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 23:20:46.210785 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:20:46.211027 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.206344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33305 (local address 127.0.98.3:36009)
0928 23:20:46.206519 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:46.206568 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:46.207198 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:46.207440 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:46.207444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:46.207454 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:46.208127 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.208133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.208748 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.208749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.209522 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.209533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.209889 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.209890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.210008 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.210469 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:20:46.210470 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:46.210498 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:20:46.210548 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:46.210708 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:46.210710 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:46.210713 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:46.210757 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:20:46.210836 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260928 23:20:46.211452 19419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33305
I20260928 23:20:46.211539 19419 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:46.211732 19419 heartbeater.cc:511] Master 127.0.98.62:33305 requested a full tablet report, sending...
I20260928 23:20:46.212116 18969 ts_manager.cc:194] Registered new tserver with Master: 722e8f4991c44e6cb26c01ef0e8366b1 (127.0.98.3:42511)
I20260928 23:20:46.212474 18969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36009
I20260928 23:20:46.218842 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:20:46.223104 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.220290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.0.98.62:33305)
0928 23:20:46.220338 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:46.220340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:46.220460 (+ 120us) server_negotiation.cc:407] Connection header received
0928 23:20:46.220511 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:46.220511 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:46.220521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:46.220603 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:46.220759 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.221227 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.221959 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.222154 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.222206 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.222844 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:46.222845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:46.222845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:46.222849 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:46.222863 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:46.222867 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:46.222947 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:46.223017 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 23:20:46.223027 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:20:46.224861 18969 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:20:46.226444 18969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:20:46.232308 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.228516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43135 (local address 127.0.0.1:37286)
0928 23:20:46.228653 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:46.228692 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:46.228751 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:46.229099 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:46.229100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:46.229102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:46.229131 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.229132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.230087 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.230088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.231332 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.231335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.232030 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.232031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.232102 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.232114 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 23:20:46.232187 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:20:46.232236 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":92,"threads_started":1}
I20260928 23:20:46.232321 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.228778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.0.98.2:43135)
0928 23:20:46.228935 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:46.228939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:46.228949 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:20:46.228986 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:46.228989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:46.229004 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:46.229108 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:46.229183 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.230009 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.231403 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.231957 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.232020 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.232111 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:46.232200 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 23:20:46.232235 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260928 23:20:46.232555 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.228516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42511 (local address 127.0.0.1:56172)
0928 23:20:46.228698 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:46.228708 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:46.228756 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:46.229029 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:46.229033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:46.229035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:46.229105 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.229107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.230349 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.230350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.231448 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.231450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.232349 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.232350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.232412 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.232417 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:20:46.232448 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 23:20:46.232489 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260928 23:20:46.232913 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.228607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.0.98.3:42511)
0928 23:20:46.228877 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:46.228882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:46.228890 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:20:46.228926 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:46.228928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:46.228942 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:46.229016 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:46.229191 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.230267 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.231524 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.232252 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.232653 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.232751 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:46.232776 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 23:20:46.232813 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":81,"threads_started":1}
I20260928 23:20:46.232968 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.228731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35021 (local address 127.0.0.1:49430)
0928 23:20:46.229107 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:46.229117 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:46.229156 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:46.229443 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:46.229444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:46.229446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:46.229482 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.229484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.230599 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.230600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.231432 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.231434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.232778 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.232779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.232832 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.232836 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:20:46.232878 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:20:46.232912 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":61,"threads_started":1}
I20260928 23:20:46.232999 19222 tablet_service.cc:1467] Processing CreateTablet for tablet b193bc3227bc41b884ef1f4d82a68592 (DEFAULT_TABLE table=test-table [id=d8ea772a73474ae38994dc34e3611bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:46.233229 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.228924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.0.98.1:35021)
0928 23:20:46.229257 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:46.229261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:46.229269 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:20:46.229303 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:46.229304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:46.229317 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:46.229382 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:46.229559 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.230531 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.231496 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.232252 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.232320 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.233094 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:46.233119 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 23:20:46.233156 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":84,"threads_started":1}
I20260928 23:20:46.233386 19222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b193bc3227bc41b884ef1f4d82a68592. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:46.233759 19091 tablet_service.cc:1467] Processing CreateTablet for tablet b193bc3227bc41b884ef1f4d82a68592 (DEFAULT_TABLE table=test-table [id=d8ea772a73474ae38994dc34e3611bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:46.234045 19091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b193bc3227bc41b884ef1f4d82a68592. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:46.236456 19433 tablet_bootstrap.cc:492] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d: Bootstrap starting.
I20260928 23:20:46.236703 19353 tablet_service.cc:1467] Processing CreateTablet for tablet b193bc3227bc41b884ef1f4d82a68592 (DEFAULT_TABLE table=test-table [id=d8ea772a73474ae38994dc34e3611bcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:46.236802 19434 tablet_bootstrap.cc:492] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750: Bootstrap starting.
I20260928 23:20:46.237022 19353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b193bc3227bc41b884ef1f4d82a68592. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:46.237133 19433 tablet_bootstrap.cc:654] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:46.237437 19434 tablet_bootstrap.cc:654] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:46.237422 19433 log.cc:824] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:46.237722 19434 log.cc:824] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:46.238178 19433 tablet_bootstrap.cc:492] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d: No bootstrap required, opened a new log
I20260928 23:20:46.238248 19433 ts_tablet_manager.cc:1397] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:20:46.238353 19434 tablet_bootstrap.cc:492] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750: No bootstrap required, opened a new log
I20260928 23:20:46.238412 19434 ts_tablet_manager.cc:1397] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:20:46.240048 19437 tablet_bootstrap.cc:492] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: Bootstrap starting.
I20260928 23:20:46.240399 19434 raft_consensus.cc:348] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } }
I20260928 23:20:46.240569 19434 raft_consensus.cc:374] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:46.240603 19434 raft_consensus.cc:729] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285eff64a8ae45e1acd2ceb868c34750, State: Initialized, Role: FOLLOWER
I20260928 23:20:46.240700 19437 tablet_bootstrap.cc:654] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:46.240721 19434 consensus_queue.cc:260] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } }
I20260928 23:20:46.240676 19433 raft_consensus.cc:348] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } }
I20260928 23:20:46.240856 19433 raft_consensus.cc:374] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:46.240897 19433 raft_consensus.cc:729] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3bf0c28af441b99a2cf4fff1cf540d, State: Initialized, Role: FOLLOWER
I20260928 23:20:46.240971 19434 ts_tablet_manager.cc:1428] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:20:46.240991 19437 log.cc:824] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:46.241003 19433 consensus_queue.cc:260] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } }
I20260928 23:20:46.241083 19157 heartbeater.cc:503] Master 127.0.98.62:33305 was elected leader, sending a full tablet report...
I20260928 23:20:46.241232 19433 ts_tablet_manager.cc:1428] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:20:46.241254 19288 heartbeater.cc:503] Master 127.0.98.62:33305 was elected leader, sending a full tablet report...
I20260928 23:20:46.241752 19437 tablet_bootstrap.cc:492] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: No bootstrap required, opened a new log
I20260928 23:20:46.241818 19437 ts_tablet_manager.cc:1397] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:20:46.243533 19437 raft_consensus.cc:348] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } }
I20260928 23:20:46.243695 19437 raft_consensus.cc:374] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:46.243716 19437 raft_consensus.cc:729] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722e8f4991c44e6cb26c01ef0e8366b1, State: Initialized, Role: FOLLOWER
I20260928 23:20:46.243815 19437 consensus_queue.cc:260] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } }
I20260928 23:20:46.244058 19437 ts_tablet_manager.cc:1428] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:20:46.244125 19419 heartbeater.cc:503] Master 127.0.98.62:33305 was elected leader, sending a full tablet report...
W20260928 23:20:46.302157 19289 tablet.cc:2369] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:20:46.350695 19439 raft_consensus.cc:482] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:46.350817 19439 raft_consensus.cc:504] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } }
I20260928 23:20:46.351248 19439 leader_election.cc:288] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 285eff64a8ae45e1acd2ceb868c34750 (127.0.98.1:35021), 722e8f4991c44e6cb26c01ef0e8366b1 (127.0.98.3:42511)
I20260928 23:20:46.354041 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.351471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40179 (local address 127.0.98.3:42511)
0928 23:20:46.351514 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:46.351516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:46.351539 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:20:46.351593 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:46.351594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:46.351603 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:46.351657 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:46.351801 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.352349 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.353344 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.353728 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.353778 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.353785 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:46.353935 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 23:20:46.353970 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:20:46.354048 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.351404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42511 (local address 127.0.98.2:40179)
0928 23:20:46.351445 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:46.351457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:46.351498 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:46.351667 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:46.351669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:46.351671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:46.351730 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.351733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.352444 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.352445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.353257 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.353260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.353799 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.353800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.353868 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.353872 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:20:46.353931 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:20:46.353980 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 23:20:46.354174 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.351404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35021 (local address 127.0.98.2:58995)
0928 23:20:46.351686 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:46.351696 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:46.351745 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:46.351872 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:46.351872 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:46.351874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:46.351936 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.351937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.352623 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.352624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.353392 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:46.353394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.353935 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.353937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:46.354012 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.354015 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:46.354067 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:20:46.354102 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":47,"thread_start_us":139,"threads_started":1}
I20260928 23:20:46.354187 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.351486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58995 (local address 127.0.98.1:35021)
0928 23:20:46.351525 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:46.351527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:46.351780 (+ 253us) server_negotiation.cc:407] Connection header received
0928 23:20:46.351809 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:46.351810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:46.351822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:46.351858 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:46.352035 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.352550 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.353470 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.353857 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.353926 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.353935 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:46.354083 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 23:20:46.354125 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:20:46.354408 19111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b193bc3227bc41b884ef1f4d82a68592" candidate_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285eff64a8ae45e1acd2ceb868c34750" is_pre_election: true
I20260928 23:20:46.354413 19373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b193bc3227bc41b884ef1f4d82a68592" candidate_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" is_pre_election: true
I20260928 23:20:46.354596 19373 raft_consensus.cc:2454] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3bf0c28af441b99a2cf4fff1cf540d in term 0.
I20260928 23:20:46.354596 19111 raft_consensus.cc:2454] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3bf0c28af441b99a2cf4fff1cf540d in term 0.
I20260928 23:20:46.354836 19175 leader_election.cc:302] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285eff64a8ae45e1acd2ceb868c34750, cd3bf0c28af441b99a2cf4fff1cf540d; no voters:
I20260928 23:20:46.354957 19439 raft_consensus.cc:2781] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:20:46.355010 19439 raft_consensus.cc:482] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:20:46.355038 19439 raft_consensus.cc:3037] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:46.355832 19439 raft_consensus.cc:504] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } }
I20260928 23:20:46.355952 19439 leader_election.cc:288] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [CANDIDATE]: Term 1 election: Requested vote from peers 285eff64a8ae45e1acd2ceb868c34750 (127.0.98.1:35021), 722e8f4991c44e6cb26c01ef0e8366b1 (127.0.98.3:42511)
I20260928 23:20:46.356079 19373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b193bc3227bc41b884ef1f4d82a68592" candidate_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722e8f4991c44e6cb26c01ef0e8366b1"
I20260928 23:20:46.356083 19111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b193bc3227bc41b884ef1f4d82a68592" candidate_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "285eff64a8ae45e1acd2ceb868c34750"
I20260928 23:20:46.356150 19373 raft_consensus.cc:3037] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:46.356168 19111 raft_consensus.cc:3037] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:46.356889 19373 raft_consensus.cc:2454] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3bf0c28af441b99a2cf4fff1cf540d in term 1.
I20260928 23:20:46.356964 19111 raft_consensus.cc:2454] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3bf0c28af441b99a2cf4fff1cf540d in term 1.
I20260928 23:20:46.357087 19176 leader_election.cc:302] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722e8f4991c44e6cb26c01ef0e8366b1, cd3bf0c28af441b99a2cf4fff1cf540d; no voters:
I20260928 23:20:46.357165 19439 raft_consensus.cc:2781] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:46.357276 19439 raft_consensus.cc:686] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 1 LEADER]: Becoming Leader. State: Replica: cd3bf0c28af441b99a2cf4fff1cf540d, State: Running, Role: LEADER
I20260928 23:20:46.357348 19439 consensus_queue.cc:237] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } }
I20260928 23:20:46.358112 18969 catalog_manager.cc:5495] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3bf0c28af441b99a2cf4fff1cf540d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3bf0c28af441b99a2cf4fff1cf540d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 } health_report { overall_health: UNKNOWN } } }
I20260928 23:20:46.376099 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.374020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.0.98.2:43135)
0928 23:20:46.374070 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:46.374072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:46.374081 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:20:46.374116 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:46.374116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:46.374136 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:46.374193 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:20:46.374309 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.374777 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.375432 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:46.375663 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:46.375710 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:46.375810 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:20:46.375894 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:20:46.375956 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:46.375992 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 23:20:46.376001 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:20:46.377856 19373 raft_consensus.cc:1261] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1 [term 1 FOLLOWER]: Refusing update from remote peer cd3bf0c28af441b99a2cf4fff1cf540d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:46.377859 19111 raft_consensus.cc:1261] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750 [term 1 FOLLOWER]: Refusing update from remote peer cd3bf0c28af441b99a2cf4fff1cf540d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:46.378201 19443 consensus_queue.cc:1035] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [LEADER]: Connected to new peer: Peer: permanent_uuid: "285eff64a8ae45e1acd2ceb868c34750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:46.378320 19439 consensus_queue.cc:1035] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [LEADER]: Connected to new peer: Peer: permanent_uuid: "722e8f4991c44e6cb26c01ef0e8366b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:46.382848 18968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:46.382929 18968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:46.383641 18968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c8247e581f84bc0872b1c7c3e9f3300: Sending DeleteTablet for 3 replicas of tablet b193bc3227bc41b884ef1f4d82a68592
I20260928 23:20:46.383948 19222 tablet_service.cc:1514] Processing DeleteTablet for tablet b193bc3227bc41b884ef1f4d82a68592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:46 UTC) from {username='slave'} at 127.0.0.1:37286
I20260928 23:20:46.383989 19091 tablet_service.cc:1514] Processing DeleteTablet for tablet b193bc3227bc41b884ef1f4d82a68592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:46 UTC) from {username='slave'} at 127.0.0.1:49430
I20260928 23:20:46.384150 19456 tablet_replica.cc:330] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d: stopping tablet replica
I20260928 23:20:46.384208 19353 tablet_service.cc:1514] Processing DeleteTablet for tablet b193bc3227bc41b884ef1f4d82a68592 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:46 UTC) from {username='slave'} at 127.0.0.1:56172
I20260928 23:20:46.384243 19456 raft_consensus.cc:2229] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:20:46.384253 19457 tablet_replica.cc:330] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750: stopping tablet replica
I20260928 23:20:46.384331 19456 raft_consensus.cc:2258] T b193bc3227bc41b884ef1f4d82a68592 P cd3bf0c28af441b99a2cf4fff1cf540d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:46.384438 19457 raft_consensus.cc:2229] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:46.384596 19457 raft_consensus.cc:2258] T b193bc3227bc41b884ef1f4d82a68592 P 285eff64a8ae45e1acd2ceb868c34750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:46.384624 19458 tablet_replica.cc:330] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: stopping tablet replica
I20260928 23:20:46.384716 19458 raft_consensus.cc:2229] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:46.384713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 19029
I20260928 23:20:46.384816 19458 raft_consensus.cc:2258] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:46.385257 19458 ts_tablet_manager.cc:1905] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:46.387102 19458 ts_tablet_manager.cc:1918] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:46.387176 19458 log.cc:1196] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/b193bc3227bc41b884ef1f4d82a68592
I20260928 23:20:46.387468 19458 ts_tablet_manager.cc:1939] T b193bc3227bc41b884ef1f4d82a68592 P 722e8f4991c44e6cb26c01ef0e8366b1: Deleting consensus metadata
I20260928 23:20:46.387883 18954 catalog_manager.cc:4841] TS 722e8f4991c44e6cb26c01ef0e8366b1 (127.0.98.3:42511): tablet b193bc3227bc41b884ef1f4d82a68592 (table test-table [id=d8ea772a73474ae38994dc34e3611bcd]) successfully deleted
W20260928 23:20:46.390456 18953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35021 (error 108)
I20260928 23:20:46.390558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 19160
I20260928 23:20:46.391727 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.391613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35021 (local address 127.0.0.1:49442)
0928 23:20:46.391652 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:46.391694 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 23:20:46.391844 18953 catalog_manager.cc:4584] TS 285eff64a8ae45e1acd2ceb868c34750 (127.0.98.1:35021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b193bc3227bc41b884ef1f4d82a68592: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35021: connect: Connection refused (error 111)
I20260928 23:20:46.394829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 19291
I20260928 23:20:46.394913 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:46.394803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43135 (local address 127.0.0.1:37304)
0928 23:20:46.394856 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:46.394882 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 23:20:46.398828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 18938
2026-09-28T23:20:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (792 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:46 dist-test-slave-djkr krb5kdc[19462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:46 dist-test-slave-djkr krb5kdc[19462](info): set up 2 sockets
Sep 28 23:20:46 dist-test-slave-djkr krb5kdc[19462](info): commencing operation
krb5kdc: starting...
W20260928 23:20:47.343034 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:47 dist-test-slave-djkr krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:20:47.453704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36627
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:47.532147 19475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:47.532357 19475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:47.532379 19475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:47.533535 19475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:47.533579 19475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:47.533592 19475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:47.533604 19475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:47.533614 19475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:47.535113 19475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:47.535347 19475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:47.535590 19475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:47.540099 19481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:47.540074 19480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:47.540083 19483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:47.540321 19475 server_base.cc:1034] running on GCE node
I20260928 23:20:47.540575 19475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:47.540834 19475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:47.541986 19475 hybrid_clock.cc:648] HybridClock initialized: now 1790637647541967 us; error 36 us; skew 500 ppm
Sep 28 23:20:47 dist-test-slave-djkr krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:47.544291 19475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:20:47.544690 19475 webserver.cc:460] Webserver started at http://127.0.98.62:45535/ using document root <none> and password file <none>
I20260928 23:20:47.544880 19475 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:47.544934 19475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:47.545068 19475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:47.546090 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "f863fa8f44414949b6444d0c48db2c66"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "67d94dec4458c06ae7e67f3cd13b2caf"
server_key_iv: "7f127a5507f054bd0ebb62a4085074be"
server_key_version: "encryptionkey@0"
I20260928 23:20:47.546408 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "f863fa8f44414949b6444d0c48db2c66"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "67d94dec4458c06ae7e67f3cd13b2caf"
server_key_iv: "7f127a5507f054bd0ebb62a4085074be"
server_key_version: "encryptionkey@0"
I20260928 23:20:47.547652 19475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:20:47.548342 19490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:47.548491 19475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:47.548573 19475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "f863fa8f44414949b6444d0c48db2c66"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "67d94dec4458c06ae7e67f3cd13b2caf"
server_key_iv: "7f127a5507f054bd0ebb62a4085074be"
server_key_version: "encryptionkey@0"
I20260928 23:20:47.548645 19475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:47.558486 19475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:47.560148 19475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:47.560276 19475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:47.563910 19475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36627
I20260928 23:20:47.563936 19542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36627 every 8 connection(s)
I20260928 23:20:47.564248 19475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:47.565040 19543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:47.567294 19543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66: Bootstrap starting.
I20260928 23:20:47.567823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 19475
I20260928 23:20:47.567893 19543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:47.567914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:47.568089 392 external_mini_cluster.cc:1428] Setting key 4df367c66e72ea40cdcc5516fb110685
I20260928 23:20:47.568145 19543 log.cc:824] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:47.568931 19543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66: No bootstrap required, opened a new log
I20260928 23:20:47.571221 19543 raft_consensus.cc:348] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f863fa8f44414949b6444d0c48db2c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260928 23:20:47.571383 19543 raft_consensus.cc:374] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:47.571416 19543 raft_consensus.cc:729] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f863fa8f44414949b6444d0c48db2c66, State: Initialized, Role: FOLLOWER
I20260928 23:20:47.571587 19543 consensus_queue.cc:260] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f863fa8f44414949b6444d0c48db2c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260928 23:20:47.571671 19543 raft_consensus.cc:388] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:47.571702 19543 raft_consensus.cc:482] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:47.571748 19543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:47.572265 19543 raft_consensus.cc:504] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f863fa8f44414949b6444d0c48db2c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260928 23:20:47.572364 19543 leader_election.cc:302] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f863fa8f44414949b6444d0c48db2c66; no voters:
I20260928 23:20:47.572505 19543 leader_election.cc:288] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:47.572537 19548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:47.572691 19548 raft_consensus.cc:686] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [term 1 LEADER]: Becoming Leader. State: Replica: f863fa8f44414949b6444d0c48db2c66, State: Running, Role: LEADER
I20260928 23:20:47.572765 19543 sys_catalog.cc:564] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:47.572829 19548 consensus_queue.cc:237] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f863fa8f44414949b6444d0c48db2c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } }
I20260928 23:20:47.573257 19549 sys_catalog.cc:455] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f863fa8f44414949b6444d0c48db2c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f863fa8f44414949b6444d0c48db2c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } } }
I20260928 23:20:47.573330 19549 sys_catalog.cc:458] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:47.573557 19550 sys_catalog.cc:455] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f863fa8f44414949b6444d0c48db2c66. Latest consensus state: current_term: 1 leader_uuid: "f863fa8f44414949b6444d0c48db2c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f863fa8f44414949b6444d0c48db2c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36627 } } }
I20260928 23:20:47.573923 19550 sys_catalog.cc:458] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:47.574079 19555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:47.574560 19555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 23:20:47 dist-test-slave-djkr krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:47.575937 19555 catalog_manager.cc:1347] Generated new cluster ID: 3f2a0a0a0651413ba81310482deab3cf
I20260928 23:20:47.575984 19555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:47.579227 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:47.568538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.0.98.62:36627)
0928 23:20:47.568678 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:47.568688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:47.568724 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:20:47.569278 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:47.569286 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:47.569356 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:47.569498 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:47.570041 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:47.571388 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:47.572241 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:47.572859 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:47.572969 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:47.575784 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:47.575788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:47.575789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:47.575800 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:47.577722 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:47.578102 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:47.578105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:47.578105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:47.578201 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:47.578382 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:47.578382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:47.578383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:47.578642 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:47.578786 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:47.578887 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 23:20:47.579009 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260928 23:20:47.591928 19555 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:47.592720 19555 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:47.596308 19555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66: Generated new TSK 0
I20260928 23:20:47.596489 19555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:20:47.622378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:47.701984 19569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:47.702178 19569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:47.702209 19569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:47.703341 19569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:47.703379 19569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:47.703413 19569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:20:47.704921 19569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:47.705149 19569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:47.705349 19569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:47.710124 19577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:47.710139 19569 server_base.cc:1034] running on GCE node
W20260928 23:20:47.710089 19575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:47.710151 19574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:47.710448 19569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:47.710670 19569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:47.711797 19569 hybrid_clock.cc:648] HybridClock initialized: now 1790637647711791 us; error 28 us; skew 500 ppm
Sep 28 23:20:47 dist-test-slave-djkr krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:47.714090 19569 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:20:47.714493 19569 webserver.cc:460] Webserver started at http://127.0.98.1:38349/ using document root <none> and password file <none>
I20260928 23:20:47.714694 19569 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:47.714740 19569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:47.714860 19569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:47.715929 19569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "d1577bb97c4f4cfe839714ffe31e493c"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "4dcef9aeceabe85281ab8d761d9703bc"
server_key_iv: "3f3604586b91a5bfdab628d5025f0ece"
server_key_version: "encryptionkey@0"
I20260928 23:20:47.716243 19569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d1577bb97c4f4cfe839714ffe31e493c"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "4dcef9aeceabe85281ab8d761d9703bc"
server_key_iv: "3f3604586b91a5bfdab628d5025f0ece"
server_key_version: "encryptionkey@0"
I20260928 23:20:47.717445 19569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:20:47.718099 19584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:47.718255 19569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:47.718333 19569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "d1577bb97c4f4cfe839714ffe31e493c"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "4dcef9aeceabe85281ab8d761d9703bc"
server_key_iv: "3f3604586b91a5bfdab628d5025f0ece"
server_key_version: "encryptionkey@0"
I20260928 23:20:47.718391 19569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:47.736497 19569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:47.738349 19569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:47.738481 19569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:47.738683 19569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:47.738969 19569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:47.739001 19569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:47.739029 19569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:47.739053 19569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:47.744390 19569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33625
I20260928 23:20:47.744446 19697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33625 every 8 connection(s)
I20260928 23:20:47.744748 19569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:20:47.747154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 19569
I20260928 23:20:47.747260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:20:47.747453 392 external_mini_cluster.cc:1428] Setting key 67e4d384e481c278ab81a75c37bd2996
Sep 28 23:20:47 dist-test-slave-djkr krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:47.754772 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:47.745468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50057 (local address 127.0.98.62:36627)
0928 23:20:47.745562 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:47.745565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:47.745751 (+ 186us) server_negotiation.cc:407] Connection header received
0928 23:20:47.746681 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:47.746682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:47.746695 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:47.746745 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:47.747582 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:47.748329 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:47.749774 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:47.749989 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:47.750060 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:47.752364 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:47.752365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:47.752366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:47.752373 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:47.753569 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:47.753925 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:47.753926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:47.753927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:47.753981 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:47.754192 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:47.754193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:47.754193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:47.754301 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:47.754354 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:47.754599 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 23:20:47.754682 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 23:20:47.754938 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:47.745397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.1:50057)
0928 23:20:47.745616 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:47.745662 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:47.746565 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:47.746821 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:47.746825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:47.747164 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:47.747448 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:47.747455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:47.748434 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:47.748435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:47.749697 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:47.749700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:47.750074 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:47.750075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:47.750207 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:47.750684 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:47.750692 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:47.752283 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:47.753641 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:47.753647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:47.753649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:47.753862 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:47.754034 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:47.754035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:47.754035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:47.754135 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:47.754353 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:47.754360 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:47.754489 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 23:20:47.754575 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:20:47.754733 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:20:47.755368 19698 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36627
I20260928 23:20:47.755457 19698 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:47.755656 19698 heartbeater.cc:511] Master 127.0.98.62:36627 requested a full tablet report, sending...
I20260928 23:20:47.756204 19507 ts_manager.cc:194] Registered new tserver with Master: d1577bb97c4f4cfe839714ffe31e493c (127.0.98.1:33625)
I20260928 23:20:47.756790 19507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:20:47.770887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:47.850063 19703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:47.850263 19703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:47.850284 19703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:47.851416 19703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:47.851455 19703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:47.851486 19703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:20:47.853024 19703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:47.853214 19703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:47.853381 19703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:47.858085 19711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:47.858085 19708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:47.858175 19709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:47.858230 19703 server_base.cc:1034] running on GCE node
I20260928 23:20:47.858515 19703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:47.858750 19703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:47.859892 19703 hybrid_clock.cc:648] HybridClock initialized: now 1790637647859874 us; error 28 us; skew 500 ppm
Sep 28 23:20:47 dist-test-slave-djkr krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:47.862170 19703 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:20:47.862560 19703 webserver.cc:460] Webserver started at http://127.0.98.2:34543/ using document root <none> and password file <none>
I20260928 23:20:47.862764 19703 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:47.862810 19703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:47.862931 19703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:47.863973 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "6c70ad6b85a04bc381f91a6c7f5afa74"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "ee0f32a0cc7b5eb7671d481eb5110d96"
server_key_iv: "ced844131c38162bae2771b53cd3d76c"
server_key_version: "encryptionkey@0"
I20260928 23:20:47.864291 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6c70ad6b85a04bc381f91a6c7f5afa74"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "ee0f32a0cc7b5eb7671d481eb5110d96"
server_key_iv: "ced844131c38162bae2771b53cd3d76c"
server_key_version: "encryptionkey@0"
I20260928 23:20:47.865516 19703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:20:47.866226 19718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:47.866355 19703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:47.866434 19703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "6c70ad6b85a04bc381f91a6c7f5afa74"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "ee0f32a0cc7b5eb7671d481eb5110d96"
server_key_iv: "ced844131c38162bae2771b53cd3d76c"
server_key_version: "encryptionkey@0"
I20260928 23:20:47.866490 19703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:47.874200 19703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:47.876219 19703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:47.876341 19703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:47.876526 19703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:47.876808 19703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:47.876840 19703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:47.876866 19703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:47.876883 19703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:47.882400 19703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33279
I20260928 23:20:47.882455 19831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33279 every 8 connection(s)
I20260928 23:20:47.882728 19703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:20:47.885130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 19703
I20260928 23:20:47.885212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:20:47.885373 392 external_mini_cluster.cc:1428] Setting key c425188ae651749d4d3762349f3b27bc
Sep 28 23:20:47 dist-test-slave-djkr krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:47.891716 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:47.883362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48675 (local address 127.0.98.62:36627)
0928 23:20:47.883405 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:47.883407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:47.883593 (+ 186us) server_negotiation.cc:407] Connection header received
0928 23:20:47.884494 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:47.884495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:47.884507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:47.884559 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:47.885136 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:47.885768 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:47.886769 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:47.886989 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:47.887055 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:47.889079 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:47.889081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:47.889081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:47.889088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:47.890371 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:47.890829 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:47.890830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:47.890831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:47.890885 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:47.891106 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:47.891107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:47.891107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:47.891214 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:47.891270 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:47.891525 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 23:20:47.891630 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:20:47.891871 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:47.883335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.2:48675)
0928 23:20:47.883466 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:47.883503 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:47.884387 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:47.884624 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:47.884627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:47.884870 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:47.885053 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:47.885058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:47.885845 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:47.885846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:47.886694 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:47.886697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:47.887096 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:47.887097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:47.887222 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:47.887667 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:47.887674 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:47.888949 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:47.890533 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:47.890539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:47.890541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:47.890750 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:47.890938 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:47.890938 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:47.890939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:47.891050 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:47.891266 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:47.891271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:47.891411 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 23:20:47.891495 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:20:47.891668 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:20:47.892307 19832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36627
I20260928 23:20:47.892400 19832 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:47.892575 19832 heartbeater.cc:511] Master 127.0.98.62:36627 requested a full tablet report, sending...
I20260928 23:20:47.892870 19507 ts_manager.cc:194] Registered new tserver with Master: 6c70ad6b85a04bc381f91a6c7f5afa74 (127.0.98.2:33279)
I20260928 23:20:47.893214 19507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:20:47.907871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:47.985855 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:47.986061 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:47.986083 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:47.987250 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:47.987291 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:47.987317 19837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:20:47.988788 19837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:47.988996 19837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:47.989209 19837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:47.993772 19845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:47.993816 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:47.993786 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:47.994014 19837 server_base.cc:1034] running on GCE node
I20260928 23:20:47.994232 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:47.994431 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:47.995608 19837 hybrid_clock.cc:648] HybridClock initialized: now 1790637647995593 us; error 28 us; skew 500 ppm
Sep 28 23:20:47 dist-test-slave-djkr krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:47.997885 19837 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:20:47.998283 19837 webserver.cc:460] Webserver started at http://127.0.98.3:37987/ using document root <none> and password file <none>
I20260928 23:20:47.998474 19837 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:47.998521 19837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:47.998641 19837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:47.999699 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "59e179fe9aa24b718247b2b448fbbe27"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "b37f0eff903f2c3f856abf9f03eedf79"
server_key_iv: "95002cc2aba37056373bf8f6552ddf31"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.000016 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "59e179fe9aa24b718247b2b448fbbe27"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "b37f0eff903f2c3f856abf9f03eedf79"
server_key_iv: "95002cc2aba37056373bf8f6552ddf31"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.001217 19837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:48.001900 19852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.002063 19837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:48.002140 19837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "59e179fe9aa24b718247b2b448fbbe27"
format_stamp: "Formatted at 2026-09-28 23:20:47 on dist-test-slave-djkr"
server_key: "b37f0eff903f2c3f856abf9f03eedf79"
server_key_iv: "95002cc2aba37056373bf8f6552ddf31"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.002199 19837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:48.031809 19837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:48.033772 19837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:48.033911 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:48.034130 19837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:48.034427 19837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:48.034459 19837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.034489 19837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:48.034507 19837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.039742 19837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43883
I20260928 23:20:48.039804 19965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43883 every 8 connection(s)
I20260928 23:20:48.040086 19837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:20:48.042132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 19837
I20260928 23:20:48.042224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:20:48.042387 392 external_mini_cluster.cc:1428] Setting key 995524d5ba150615af4095b529c4f553
Sep 28 23:20:48 dist-test-slave-djkr krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:20:48.048753 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.040745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53601 (local address 127.0.98.62:36627)
0928 23:20:48.040805 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.040807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.040986 (+ 179us) server_negotiation.cc:407] Connection header received
0928 23:20:48.041901 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.041903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.041915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.041965 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:48.042691 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.043269 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.044278 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.044458 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.044517 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.046402 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:48.046404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:48.046404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:48.046410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:48.047631 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:48.047972 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:48.047973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:48.047973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:48.048023 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:48.048215 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:48.048215 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:48.048215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:48.048309 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:48.048354 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.048617 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.048687 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:20:48.048880 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.040696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36627 (local address 127.0.98.3:53601)
0928 23:20:48.040870 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.040905 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.041794 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.042037 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.042041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.042317 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:48.042545 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.042551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.043355 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.043356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.044211 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.044213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.044524 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.044525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.044616 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.045012 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:20:48.045020 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:48.046320 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:48.047692 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:48.047698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:48.047700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:48.047916 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:48.048073 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:20:48.048073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:20:48.048073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:20:48.048174 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:20:48.048358 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:48.048363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:48.048479 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.048565 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.048700 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260928 23:20:48.049278 19966 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36627
I20260928 23:20:48.049371 19966 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:48.049552 19966 heartbeater.cc:511] Master 127.0.98.62:36627 requested a full tablet report, sending...
I20260928 23:20:48.049881 19507 ts_manager.cc:194] Registered new tserver with Master: 59e179fe9aa24b718247b2b448fbbe27 (127.0.98.3:43883)
I20260928 23:20:48.050218 19507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53601
I20260928 23:20:48.054040 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:20:48.061002 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.055395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.0.98.62:36627)
0928 23:20:48.055460 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.055462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.055520 (+ 58us) server_negotiation.cc:407] Connection header received
0928 23:20:48.055615 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.055616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.055625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.055671 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:48.056192 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.056697 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.057409 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.057608 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.057658 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.058679 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:48.058680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:48.058680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:48.058687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:48.059984 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:48.060292 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:48.060293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:48.060294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:48.060344 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:48.060504 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:48.060504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:48.060505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:48.060639 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:48.060728 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.060841 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.060912 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:20:48.062420 19507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:20:48.064077 19507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:20:48.069917 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.066055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.0.98.3:43883)
0928 23:20:48.066410 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.066415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.066426 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:20:48.066482 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.066485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.066504 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.066748 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:48.066834 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.067570 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.068943 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.069547 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.069607 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.069668 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.069797 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.069837 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
I20260928 23:20:48.069953 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.065908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43883 (local address 127.0.0.1:52502)
0928 23:20:48.066065 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.066356 (+ 291us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.066411 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.066608 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.066611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.066613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:48.066678 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.066679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.067652 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.067653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.068854 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.068856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.069619 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.069620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.069698 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.069710 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.069779 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.069831 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":56,"threads_started":1}
I20260928 23:20:48.070103 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.065929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33625 (local address 127.0.0.1:55132)
0928 23:20:48.066222 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.066262 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.066320 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.066612 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.066613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.066615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:48.066839 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.066841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.068119 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.068120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.068978 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.068981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.069852 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.069853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.069918 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.069923 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.069990 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.070032 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":56,"threads_started":1}
I20260928 23:20:48.070322 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.066102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33279 (local address 127.0.0.1:46116)
0928 23:20:48.066494 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.066501 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.066548 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.066716 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.066717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.066718 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:48.066749 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.066750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.068017 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.068018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.069173 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.069175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.070116 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.070117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.070181 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.070184 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.070227 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.070262 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":73,"threads_started":1}
I20260928 23:20:48.070415 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.066053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.0.98.1:33625)
0928 23:20:48.066459 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.066462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.066471 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:20:48.066506 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.066507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.066519 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.066910 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:48.066992 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.068036 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.069045 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.069753 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.070223 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.070283 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.070306 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.070342 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":230,"threads_started":1}
I20260928 23:20:48.070748 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.066213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.0.98.2:33279)
0928 23:20:48.066414 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.066419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.066560 (+ 141us) server_negotiation.cc:407] Connection header received
0928 23:20:48.066602 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.066604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.066617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.066719 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:48.066802 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.067906 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.069247 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.069934 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.070035 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.070094 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.070465 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.070505 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
I20260928 23:20:48.070864 19900 tablet_service.cc:1467] Processing CreateTablet for tablet eb68daf80ed84df884a1a6fc51e831b8 (DEFAULT_TABLE table=test-table [id=ac20459394504231916f3cd65c24853f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:48.071112 19632 tablet_service.cc:1467] Processing CreateTablet for tablet eb68daf80ed84df884a1a6fc51e831b8 (DEFAULT_TABLE table=test-table [id=ac20459394504231916f3cd65c24853f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:48.071202 19900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb68daf80ed84df884a1a6fc51e831b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:48.071355 19632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb68daf80ed84df884a1a6fc51e831b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:48.071372 19766 tablet_service.cc:1467] Processing CreateTablet for tablet eb68daf80ed84df884a1a6fc51e831b8 (DEFAULT_TABLE table=test-table [id=ac20459394504231916f3cd65c24853f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:48.071657 19766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb68daf80ed84df884a1a6fc51e831b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:48.074059 19980 tablet_bootstrap.cc:492] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c: Bootstrap starting.
I20260928 23:20:48.074412 19981 tablet_bootstrap.cc:492] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: Bootstrap starting.
I20260928 23:20:48.074735 19980 tablet_bootstrap.cc:654] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:48.074807 19982 tablet_bootstrap.cc:492] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74: Bootstrap starting.
I20260928 23:20:48.074991 19981 tablet_bootstrap.cc:654] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:48.075353 19982 tablet_bootstrap.cc:654] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:48.075351 19980 log.cc:824] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:48.075493 19981 log.cc:824] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:48.075914 19982 log.cc:824] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:48.076067 19980 tablet_bootstrap.cc:492] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c: No bootstrap required, opened a new log
I20260928 23:20:48.076128 19980 ts_tablet_manager.cc:1397] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:20:48.076267 19981 tablet_bootstrap.cc:492] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: No bootstrap required, opened a new log
I20260928 23:20:48.076309 19981 ts_tablet_manager.cc:1397] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:20:48.077497 19980 raft_consensus.cc:348] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 } } peers { permanent_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33279 } } peers { permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 } }
I20260928 23:20:48.077618 19980 raft_consensus.cc:374] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:48.077636 19980 raft_consensus.cc:729] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1577bb97c4f4cfe839714ffe31e493c, State: Initialized, Role: FOLLOWER
I20260928 23:20:48.077728 19980 consensus_queue.cc:260] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 } } peers { permanent_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33279 } } peers { permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 } }
I20260928 23:20:48.077677 19981 raft_consensus.cc:348] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 } } peers { permanent_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33279 } } peers { permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 } }
I20260928 23:20:48.077793 19981 raft_consensus.cc:374] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:48.077813 19981 raft_consensus.cc:729] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e179fe9aa24b718247b2b448fbbe27, State: Initialized, Role: FOLLOWER
I20260928 23:20:48.077898 19981 consensus_queue.cc:260] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 } } peers { permanent_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33279 } } peers { permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 } }
I20260928 23:20:48.077957 19980 ts_tablet_manager.cc:1428] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:20:48.077981 19698 heartbeater.cc:503] Master 127.0.98.62:36627 was elected leader, sending a full tablet report...
I20260928 23:20:48.078161 19981 ts_tablet_manager.cc:1428] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:20:48.078267 19966 heartbeater.cc:503] Master 127.0.98.62:36627 was elected leader, sending a full tablet report...
I20260928 23:20:48.078744 19982 tablet_bootstrap.cc:492] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74: No bootstrap required, opened a new log
I20260928 23:20:48.078838 19982 ts_tablet_manager.cc:1397] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:20:48.080319 19982 raft_consensus.cc:348] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 } } peers { permanent_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33279 } } peers { permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 } }
I20260928 23:20:48.080492 19982 raft_consensus.cc:374] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:48.080539 19982 raft_consensus.cc:729] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70ad6b85a04bc381f91a6c7f5afa74, State: Initialized, Role: FOLLOWER
I20260928 23:20:48.080644 19982 consensus_queue.cc:260] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 } } peers { permanent_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33279 } } peers { permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 } }
I20260928 23:20:48.080864 19982 ts_tablet_manager.cc:1428] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:20:48.080885 19832 heartbeater.cc:503] Master 127.0.98.62:36627 was elected leader, sending a full tablet report...
W20260928 23:20:48.133281 19833 tablet.cc:2369] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:20:48.181223 19988 raft_consensus.cc:482] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:48.181376 19988 raft_consensus.cc:504] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 } } peers { permanent_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33279 } } peers { permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 } }
I20260928 23:20:48.181782 19988 leader_election.cc:288] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59e179fe9aa24b718247b2b448fbbe27 (127.0.98.3:43883), d1577bb97c4f4cfe839714ffe31e493c (127.0.98.1:33625)
I20260928 23:20:48.185609 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.182106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39845 (local address 127.0.98.1:33625)
0928 23:20:48.182160 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.182162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.182172 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:20:48.182196 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.182197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.182206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.182259 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:48.182417 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.183170 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.184703 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.185280 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.185351 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.185360 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.185503 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.185540 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:20:48.185609 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.182032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42341 (local address 127.0.98.3:43883)
0928 23:20:48.182079 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.182081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.182273 (+ 192us) server_negotiation.cc:407] Connection header received
0928 23:20:48.182302 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.182303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.182311 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.182353 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:48.182481 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.183183 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.184703 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.185280 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.185351 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.185360 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.185503 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.185540 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:20:48.185645 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.181969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33625 (local address 127.0.98.2:39845)
0928 23:20:48.182008 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.182020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.182067 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.182277 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.182278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.182279 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:48.182348 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.182350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.183279 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.183280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.184607 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.184610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.185356 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.185357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.185441 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.185443 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.185491 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.185538 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":46}
I20260928 23:20:48.185652 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.181973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43883 (local address 127.0.98.2:42341)
0928 23:20:48.182201 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.182209 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.182259 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.182368 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.182369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.182370 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:48.182424 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.182425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.183279 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.183280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.184607 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.184610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.185357 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.185357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.185433 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.185437 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.185491 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.185570 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":123,"thread_start_us":116,"threads_started":1}
I20260928 23:20:48.185986 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb68daf80ed84df884a1a6fc51e831b8" candidate_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1577bb97c4f4cfe839714ffe31e493c" is_pre_election: true
I20260928 23:20:48.186151 19652 raft_consensus.cc:2454] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c70ad6b85a04bc381f91a6c7f5afa74 in term 0.
I20260928 23:20:48.186245 19920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb68daf80ed84df884a1a6fc51e831b8" candidate_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e179fe9aa24b718247b2b448fbbe27" is_pre_election: true
I20260928 23:20:48.186394 19920 raft_consensus.cc:2454] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c70ad6b85a04bc381f91a6c7f5afa74 in term 0.
I20260928 23:20:48.186383 19720 leader_election.cc:302] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70ad6b85a04bc381f91a6c7f5afa74, d1577bb97c4f4cfe839714ffe31e493c; no voters:
I20260928 23:20:48.186491 19988 raft_consensus.cc:2781] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:20:48.186554 19988 raft_consensus.cc:482] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:20:48.186578 19988 raft_consensus.cc:3037] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:48.187214 19988 raft_consensus.cc:504] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 } } peers { permanent_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33279 } } peers { permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 } }
I20260928 23:20:48.187336 19988 leader_election.cc:288] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [CANDIDATE]: Term 1 election: Requested vote from peers 59e179fe9aa24b718247b2b448fbbe27 (127.0.98.3:43883), d1577bb97c4f4cfe839714ffe31e493c (127.0.98.1:33625)
I20260928 23:20:48.187490 19920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb68daf80ed84df884a1a6fc51e831b8" candidate_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e179fe9aa24b718247b2b448fbbe27"
I20260928 23:20:48.187502 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb68daf80ed84df884a1a6fc51e831b8" candidate_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1577bb97c4f4cfe839714ffe31e493c"
I20260928 23:20:48.187589 19920 raft_consensus.cc:3037] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:48.187606 19652 raft_consensus.cc:3037] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:48.188277 19652 raft_consensus.cc:2454] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c70ad6b85a04bc381f91a6c7f5afa74 in term 1.
I20260928 23:20:48.188275 19920 raft_consensus.cc:2454] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c70ad6b85a04bc381f91a6c7f5afa74 in term 1.
I20260928 23:20:48.188426 19720 leader_election.cc:302] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70ad6b85a04bc381f91a6c7f5afa74, d1577bb97c4f4cfe839714ffe31e493c; no voters:
I20260928 23:20:48.188526 19988 raft_consensus.cc:2781] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:48.188656 19988 raft_consensus.cc:686] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 1 LEADER]: Becoming Leader. State: Replica: 6c70ad6b85a04bc381f91a6c7f5afa74, State: Running, Role: LEADER
I20260928 23:20:48.188741 19988 consensus_queue.cc:237] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 } } peers { permanent_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33279 } } peers { permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 } }
I20260928 23:20:48.189509 19506 catalog_manager.cc:5495] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c70ad6b85a04bc381f91a6c7f5afa74 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c70ad6b85a04bc381f91a6c7f5afa74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 } health_report { overall_health: UNKNOWN } } }
I20260928 23:20:48.217360 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.215137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.0.98.2:33279)
0928 23:20:48.215195 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.215196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.215207 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:20:48.215250 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.215251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.215270 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.215330 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:20:48.215455 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.215964 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.216680 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.216872 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.216930 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.217061 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:20:48.217169 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:20:48.217232 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.217244 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.217264 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:20:48.219132 19652 raft_consensus.cc:1261] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c [term 1 FOLLOWER]: Refusing update from remote peer 6c70ad6b85a04bc381f91a6c7f5afa74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:48.219132 19920 raft_consensus.cc:1261] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27 [term 1 FOLLOWER]: Refusing update from remote peer 6c70ad6b85a04bc381f91a6c7f5afa74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:48.219465 19988 consensus_queue.cc:1035] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59e179fe9aa24b718247b2b448fbbe27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:48.219623 19990 consensus_queue.cc:1035] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1577bb97c4f4cfe839714ffe31e493c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:48.222095 19994 mvcc.cc:205] Tried to move back new op lower bound from 7334451807103094784 to 7334451806981558272. Current Snapshot: MvccSnapshot[applied={T|T < 7334451807103094784}]
I20260928 23:20:48.222378 19993 mvcc.cc:205] Tried to move back new op lower bound from 7334451807103094784 to 7334451806981558272. Current Snapshot: MvccSnapshot[applied={T|T < 7334451807103094784 or (T in {7334451807103094784})}]
I20260928 23:20:48.222579 19995 mvcc.cc:205] Tried to move back new op lower bound from 7334451807103094784 to 7334451806981558272. Current Snapshot: MvccSnapshot[applied={T|T < 7334451807103094784}]
I20260928 23:20:48.224169 19507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:48.224254 19507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:48.224964 19507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f863fa8f44414949b6444d0c48db2c66: Sending DeleteTablet for 3 replicas of tablet eb68daf80ed84df884a1a6fc51e831b8
I20260928 23:20:48.225301 19632 tablet_service.cc:1514] Processing DeleteTablet for tablet eb68daf80ed84df884a1a6fc51e831b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55132
I20260928 23:20:48.225517 20003 tablet_replica.cc:330] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c: stopping tablet replica
I20260928 23:20:48.225519 19900 tablet_service.cc:1514] Processing DeleteTablet for tablet eb68daf80ed84df884a1a6fc51e831b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52502
I20260928 23:20:48.225616 20003 raft_consensus.cc:2229] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:48.225725 20003 raft_consensus.cc:2258] T eb68daf80ed84df884a1a6fc51e831b8 P d1577bb97c4f4cfe839714ffe31e493c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:48.225744 20004 tablet_replica.cc:330] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: stopping tablet replica
I20260928 23:20:48.225831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 19569
I20260928 23:20:48.225865 20004 raft_consensus.cc:2229] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:48.225957 20004 raft_consensus.cc:2258] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:48.226636 20004 ts_tablet_manager.cc:1905] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:48.228281 20004 ts_tablet_manager.cc:1918] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:48.228389 20004 log.cc:1196] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/eb68daf80ed84df884a1a6fc51e831b8
I20260928 23:20:48.228679 20004 ts_tablet_manager.cc:1939] T eb68daf80ed84df884a1a6fc51e831b8 P 59e179fe9aa24b718247b2b448fbbe27: Deleting consensus metadata
I20260928 23:20:48.229126 19491 catalog_manager.cc:4841] TS 59e179fe9aa24b718247b2b448fbbe27 (127.0.98.3:43883): tablet eb68daf80ed84df884a1a6fc51e831b8 (table test-table [id=ac20459394504231916f3cd65c24853f]) successfully deleted
W20260928 23:20:48.230897 19492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33625 (error 108)
I20260928 23:20:48.231141 19766 tablet_service.cc:1514] Processing DeleteTablet for tablet eb68daf80ed84df884a1a6fc51e831b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46116
I20260928 23:20:48.231360 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.231242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33625 (local address 127.0.0.1:55144)
0928 23:20:48.231285 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.231328 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 23:20:48.231366 20006 tablet_replica.cc:330] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74: stopping tablet replica
I20260928 23:20:48.231482 20006 raft_consensus.cc:2229] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 1 LEADER]: Raft consensus shutting down.
W20260928 23:20:48.231472 19492 catalog_manager.cc:4584] TS d1577bb97c4f4cfe839714ffe31e493c (127.0.98.1:33625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb68daf80ed84df884a1a6fc51e831b8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33625: connect: Connection refused (error 111)
I20260928 23:20:48.231642 20006 raft_consensus.cc:2258] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:48.231998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 19703
I20260928 23:20:48.232092 20006 ts_tablet_manager.cc:1905] T eb68daf80ed84df884a1a6fc51e831b8 P 6c70ad6b85a04bc381f91a6c7f5afa74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:48.236181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 19837
I20260928 23:20:48.236261 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.236150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33279 (local address 127.0.0.1:46140)
0928 23:20:48.236199 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.236229 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 23:20:48.239966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 19475
2026-09-28T23:20:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1841 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T23:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:48Z Disabled control of system clock
I20260928 23:20:48.265208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38849
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:20:48.342741 20010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:48.342936 20010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:48.342954 20010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:48.344023 20010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:48.344055 20010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:48.344066 20010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:48.344077 20010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:48.344087 20010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:48.345350 20010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:48.345536 20010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:48.345755 20010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:48.350093 20015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:48.350162 20010 server_base.cc:1034] running on GCE node
W20260928 23:20:48.350118 20018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:48.350118 20016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:48.350574 20010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:48.350844 20010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:48.351971 20010 hybrid_clock.cc:648] HybridClock initialized: now 1790637648351963 us; error 34 us; skew 500 ppm
I20260928 23:20:48.353152 20010 webserver.cc:460] Webserver started at http://127.0.98.62:42725/ using document root <none> and password file <none>
I20260928 23:20:48.353363 20010 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:48.353405 20010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:48.353525 20010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:48.354547 20010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "1acb4aaa8ebb4af9889ba5728e174ece"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "e0989ac226fbcee9455b1d1bc0e853bc"
server_key_iv: "beb49a1ddfc92d89412271d896098f90"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.354856 20010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "1acb4aaa8ebb4af9889ba5728e174ece"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "e0989ac226fbcee9455b1d1bc0e853bc"
server_key_iv: "beb49a1ddfc92d89412271d896098f90"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.356081 20010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:20:48.356712 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.356851 20010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:48.356923 20010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "1acb4aaa8ebb4af9889ba5728e174ece"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "e0989ac226fbcee9455b1d1bc0e853bc"
server_key_iv: "beb49a1ddfc92d89412271d896098f90"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.356985 20010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:48.364162 20010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:48.364423 20010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:48.364557 20010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:48.368150 20010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38849
I20260928 23:20:48.368180 20076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38849 every 8 connection(s)
I20260928 23:20:48.368459 20010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:48.368721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 20010
I20260928 23:20:48.368820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:48.369004 392 external_mini_cluster.cc:1428] Setting key cab2b0e80cd1e4c36f713731eac27996
I20260928 23:20:48.369444 20077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:48.372030 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece: Bootstrap starting.
I20260928 23:20:48.372551 20077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:48.372809 20077 log.cc:824] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:48.373454 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece: No bootstrap required, opened a new log
I20260928 23:20:48.373901 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.369451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.0.98.62:38849)
0928 23:20:48.369657 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.369666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.369693 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:20:48.370125 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.370133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.370216 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.370333 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:48.370526 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.371388 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.372178 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.372573 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.372689 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.373309 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:48.373313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:48.373313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:48.373454 (+ 141us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:48.373484 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:48.373494 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:48.373544 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.373675 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.373752 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260928 23:20:48.375370 20077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acb4aaa8ebb4af9889ba5728e174ece" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } }
I20260928 23:20:48.375495 20077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:48.375527 20077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1acb4aaa8ebb4af9889ba5728e174ece, State: Initialized, Role: FOLLOWER
I20260928 23:20:48.375669 20077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acb4aaa8ebb4af9889ba5728e174ece" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } }
I20260928 23:20:48.375743 20077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:48.375772 20077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:48.375818 20077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:48.376304 20077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acb4aaa8ebb4af9889ba5728e174ece" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } }
I20260928 23:20:48.376392 20077 leader_election.cc:302] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1acb4aaa8ebb4af9889ba5728e174ece; no voters:
I20260928 23:20:48.376544 20077 leader_election.cc:288] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:48.376573 20082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:48.376703 20082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [term 1 LEADER]: Becoming Leader. State: Replica: 1acb4aaa8ebb4af9889ba5728e174ece, State: Running, Role: LEADER
I20260928 23:20:48.376784 20077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:48.376823 20082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acb4aaa8ebb4af9889ba5728e174ece" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } }
I20260928 23:20:48.377246 20084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1acb4aaa8ebb4af9889ba5728e174ece. Latest consensus state: current_term: 1 leader_uuid: "1acb4aaa8ebb4af9889ba5728e174ece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acb4aaa8ebb4af9889ba5728e174ece" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } } }
I20260928 23:20:48.377477 20084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:48.377415 20083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1acb4aaa8ebb4af9889ba5728e174ece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acb4aaa8ebb4af9889ba5728e174ece" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } } }
I20260928 23:20:48.377692 20083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:48.377785 20091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:48.378183 20091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:48.379666 20091 catalog_manager.cc:1347] Generated new cluster ID: f3f9999643b2424eb016f2ccf107a1ce
I20260928 23:20:48.379712 20091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:48.393093 20091 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:48.393570 20091 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:48.397039 20091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece: Generated new TSK 0
I20260928 23:20:48.397200 20091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:48.404794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--builtin_ntp_servers=127.0.98.20:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:20:48.482664 20101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:48.482872 20101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:48.482901 20101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:48.484124 20101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:48.484164 20101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:48.484198 20101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:20:48.485581 20101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:48.485808 20101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:48.486083 20101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:48.490804 20106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:48.490906 20101 server_base.cc:1034] running on GCE node
W20260928 23:20:48.490784 20107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:48.490803 20109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:48.491250 20101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:48.491494 20101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:48.492668 20101 hybrid_clock.cc:648] HybridClock initialized: now 1790637648492652 us; error 32 us; skew 500 ppm
I20260928 23:20:48.493950 20101 webserver.cc:460] Webserver started at http://127.0.98.1:35221/ using document root <none> and password file <none>
I20260928 23:20:48.494222 20101 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:48.494277 20101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:48.494412 20101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:48.495769 20101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/data/instance:
uuid: "f49d267e306b41d08461e25605e2e639"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "63fdc04f33c8bb0cbdcd93abfdbaafd0"
server_key_iv: "b6c38fe206594814fe149372add5230a"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.496141 20101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f49d267e306b41d08461e25605e2e639"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "63fdc04f33c8bb0cbdcd93abfdbaafd0"
server_key_iv: "b6c38fe206594814fe149372add5230a"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.497576 20101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:20:48.498395 20115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.498551 20101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:48.498631 20101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/wal
uuid: "f49d267e306b41d08461e25605e2e639"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "63fdc04f33c8bb0cbdcd93abfdbaafd0"
server_key_iv: "b6c38fe206594814fe149372add5230a"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.498703 20101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:48.517741 20101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:48.518033 20101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:48.518168 20101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:48.518392 20101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:48.518707 20101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:48.518740 20101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.518769 20101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:48.518785 20101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.524242 20101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37079
I20260928 23:20:48.524286 20228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37079 every 8 connection(s)
I20260928 23:20:48.524624 20101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:20:48.529477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 20101
I20260928 23:20:48.529500 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.525385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49927 (local address 127.0.98.62:38849)
0928 23:20:48.525442 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.525444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.525651 (+ 207us) server_negotiation.cc:407] Connection header received
0928 23:20:48.526112 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.526112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.526124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.526189 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:48.526833 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.527324 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.528353 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.528534 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.528583 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.529271 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:48.529272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:48.529272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:48.529276 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:48.529289 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:48.529292 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:48.529329 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.529406 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.529440 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:20:48.529587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:20:48.529733 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.525312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38849 (local address 127.0.98.1:49927)
0928 23:20:48.525531 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.525566 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.526037 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.526262 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.526265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.526274 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:48.526765 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.526769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.527420 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.527421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.528284 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.528288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.528594 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.528595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.528703 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.529154 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:20:48.529155 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:48.529182 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:20:48.529204 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:48.529333 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:48.529338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:48.529341 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.529403 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.529511 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260928 23:20:48.529780 392 external_mini_cluster.cc:1428] Setting key 49d7ea6519e2912697e7b981d79085fa
I20260928 23:20:48.530202 20229 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38849
I20260928 23:20:48.530289 20229 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:48.530485 20229 heartbeater.cc:511] Master 127.0.98.62:38849 requested a full tablet report, sending...
I20260928 23:20:48.530869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--builtin_ntp_servers=127.0.98.20:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:20:48.531009 20040 ts_manager.cc:194] Registered new tserver with Master: f49d267e306b41d08461e25605e2e639 (127.0.98.1:37079)
I20260928 23:20:48.531628 20040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49927
W20260928 23:20:48.608500 20232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:48.608707 20232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:48.608728 20232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:48.609890 20232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:48.609931 20232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:48.609956 20232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:20:48.611331 20232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:48.611575 20232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:48.611857 20232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:48.616428 20240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:48.616508 20237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:48.616524 20238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:48.616528 20232 server_base.cc:1034] running on GCE node
I20260928 23:20:48.616871 20232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:48.617115 20232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:48.618283 20232 hybrid_clock.cc:648] HybridClock initialized: now 1790637648618261 us; error 32 us; skew 500 ppm
I20260928 23:20:48.619482 20232 webserver.cc:460] Webserver started at http://127.0.98.2:38171/ using document root <none> and password file <none>
I20260928 23:20:48.619741 20232 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:48.619791 20232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:48.619918 20232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:48.620939 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/data/instance:
uuid: "a3c989c7a1a344da985163d9e9360d58"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "b662299ce78b00ffa8f077be90c83166"
server_key_iv: "f7a842249947d0aad3af9938db9253a4"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.621272 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a3c989c7a1a344da985163d9e9360d58"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "b662299ce78b00ffa8f077be90c83166"
server_key_iv: "f7a842249947d0aad3af9938db9253a4"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.622426 20232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:20:48.623027 20246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.623210 20232 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:48.623296 20232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/wal
uuid: "a3c989c7a1a344da985163d9e9360d58"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "b662299ce78b00ffa8f077be90c83166"
server_key_iv: "f7a842249947d0aad3af9938db9253a4"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.623363 20232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:48.641705 20232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:48.641994 20232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:48.642117 20232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:48.642329 20232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:48.642665 20232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:48.642699 20232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.642726 20232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:48.642750 20232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.648025 20232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33611
I20260928 23:20:48.648088 20359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33611 every 8 connection(s)
I20260928 23:20:48.648368 20232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:20:48.652961 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.648998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48375 (local address 127.0.98.62:38849)
0928 23:20:48.649079 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.649081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.649262 (+ 181us) server_negotiation.cc:407] Connection header received
0928 23:20:48.649729 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.649730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.649740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.649789 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:48.650409 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.650892 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.651865 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.652038 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.652080 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.652764 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:48.652766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:48.652766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:48.652770 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:48.652785 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:48.652788 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:48.652827 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.652880 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.652900 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:20:48.653146 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.648974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38849 (local address 127.0.98.2:48375)
0928 23:20:48.649149 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.649181 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.649654 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.649843 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.649846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.649855 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:48.650326 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.650330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.650969 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.650970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.651802 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.651804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.652099 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.652100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.652208 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.652642 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:20:48.652642 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:48.652669 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:20:48.652691 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:48.652827 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:48.652829 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:48.652831 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.652874 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.652947 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260928 23:20:48.653784 20360 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38849
I20260928 23:20:48.653867 20360 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:48.654006 20360 heartbeater.cc:511] Master 127.0.98.62:38849 requested a full tablet report, sending...
I20260928 23:20:48.654358 20040 ts_manager.cc:194] Registered new tserver with Master: a3c989c7a1a344da985163d9e9360d58 (127.0.98.2:33611)
I20260928 23:20:48.654697 20040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48375
I20260928 23:20:48.655273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 20232
I20260928 23:20:48.655380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:20:48.655599 392 external_mini_cluster.cc:1428] Setting key 9c4803b6cda12ad582da5d94bae21b4c
I20260928 23:20:48.656399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--builtin_ntp_servers=127.0.98.20:44827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:20:48.734777 20363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:48.734969 20363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:48.734989 20363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:48.736090 20363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:48.736124 20363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:48.736147 20363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:20:48.737414 20363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:48.737596 20363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:48.737829 20363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:48.742528 20368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:48.742519 20371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:48.742558 20369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:48.742895 20363 server_base.cc:1034] running on GCE node
I20260928 23:20:48.743083 20363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:48.743291 20363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:48.744446 20363 hybrid_clock.cc:648] HybridClock initialized: now 1790637648744440 us; error 34 us; skew 500 ppm
I20260928 23:20:48.745642 20363 webserver.cc:460] Webserver started at http://127.0.98.3:39701/ using document root <none> and password file <none>
I20260928 23:20:48.745853 20363 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:48.745903 20363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:48.746023 20363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:48.747031 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/data/instance:
uuid: "d9a85126635a4982a7a2acb156835b54"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "0041b8ad0acd0217cc37ae49c75b2126"
server_key_iv: "15ce021b81e5c3de21a9d2346af1634c"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.747359 20363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d9a85126635a4982a7a2acb156835b54"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "0041b8ad0acd0217cc37ae49c75b2126"
server_key_iv: "15ce021b81e5c3de21a9d2346af1634c"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.748600 20363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:20:48.749243 20377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.749392 20363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:48.749467 20363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/wal
uuid: "d9a85126635a4982a7a2acb156835b54"
format_stamp: "Formatted at 2026-09-28 23:20:48 on dist-test-slave-djkr"
server_key: "0041b8ad0acd0217cc37ae49c75b2126"
server_key_iv: "15ce021b81e5c3de21a9d2346af1634c"
server_key_version: "encryptionkey@0"
I20260928 23:20:48.749531 20363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:48.764988 20363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:48.765275 20363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:48.765399 20363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:48.765599 20363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:20:48.765906 20363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:20:48.765938 20363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.765965 20363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:20:48.765988 20363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:48.771350 20363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42727
I20260928 23:20:48.771404 20490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42727 every 8 connection(s)
I20260928 23:20:48.771749 20363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:20:48.776641 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.772450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37769 (local address 127.0.98.62:38849)
0928 23:20:48.772510 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.772512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.772730 (+ 218us) server_negotiation.cc:407] Connection header received
0928 23:20:48.773191 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.773192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.773201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.773252 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:48.773901 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.774380 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.775291 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.775467 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.775513 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.776429 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:48.776431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:48.776432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:48.776437 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:48.776454 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:48.776458 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:48.776510 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.776562 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.776582 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:20:48.776850 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.772422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38849 (local address 127.0.98.3:37769)
0928 23:20:48.772616 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.772656 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.773111 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.773315 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.773318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.773326 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:20:48.773828 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.773834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.774443 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.774444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.775233 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.775235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.775539 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.775540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.775793 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.776284 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:20:48.776284 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:20:48.776314 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:20:48.776337 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:20:48.776513 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:20:48.776516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:20:48.776518 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.776558 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.776635 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260928 23:20:48.777277 20491 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38849
I20260928 23:20:48.777364 20491 heartbeater.cc:464] Registering TS with master...
I20260928 23:20:48.777503 20491 heartbeater.cc:511] Master 127.0.98.62:38849 requested a full tablet report, sending...
I20260928 23:20:48.777842 20040 ts_manager.cc:194] Registered new tserver with Master: d9a85126635a4982a7a2acb156835b54 (127.0.98.3:42727)
I20260928 23:20:48.778184 20040 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37769
I20260928 23:20:48.780646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 20363
I20260928 23:20:48.780737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:20:48.780905 392 external_mini_cluster.cc:1428] Setting key 2a6b928720e7283de61d8463ed710b0c
I20260928 23:20:48.781603 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:20:48.785567 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.782847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.0.98.62:38849)
0928 23:20:48.782902 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.782903 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.783038 (+ 135us) server_negotiation.cc:407] Connection header received
0928 23:20:48.783097 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.783098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.783108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.783153 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:48.783300 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.783814 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.784549 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.784729 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.784779 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.785360 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:48.785361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:48.785362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:48.785366 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:48.785379 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:48.785382 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:48.785421 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.785475 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.785495 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:20:48.787009 20040 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:20:48.788548 20040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:20:48.794500 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.790734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.0.98.3:42727)
0928 23:20:48.790882 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.790887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.790921 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:20:48.790960 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.790962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.790978 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.791079 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:48.791255 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.792434 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.793680 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.794147 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.794210 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.794302 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.794390 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.794423 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260928 23:20:48.794540 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.790480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33611 (local address 127.0.0.1:52066)
0928 23:20:48.790766 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.790807 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.790872 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.791246 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.791247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.791249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:48.791279 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.791280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.792570 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.792571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.793444 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.793447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.794318 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.794319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.794377 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.794381 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.794409 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.794447 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":15,"thread_start_us":89,"threads_started":1}
I20260928 23:20:48.794523 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.790476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42727 (local address 127.0.0.1:55888)
0928 23:20:48.790766 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.790807 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.790872 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.791083 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.791086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.791088 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:48.791174 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.791177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.792518 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.792519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.793607 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.793609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.794225 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.794226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.794298 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.794310 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.794367 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.794419 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":39,"thread_start_us":69,"threads_started":1}
I20260928 23:20:48.794703 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.790568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46868 (local address 127.0.98.1:37079)
0928 23:20:48.791119 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.791123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.791132 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:20:48.791176 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.791178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.791192 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.791262 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:48.791443 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.792185 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.793550 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.794224 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.794290 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.794377 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.794567 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.794609 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":98,"threads_started":1}
I20260928 23:20:48.794855 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.790476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37079 (local address 127.0.0.1:46868)
0928 23:20:48.790957 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.790964 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.790994 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.791329 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.791330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.791332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:48.791374 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.791375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.792258 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.792259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.793485 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.793487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.794405 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.794406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.794453 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.794456 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.794498 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.794530 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":10,"thread_start_us":77,"threads_started":1}
I20260928 23:20:48.795095 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.790902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.0.98.2:33611)
0928 23:20:48.791079 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.791084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.791092 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:20:48.791124 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.791126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.791141 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.791344 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:48.791405 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.792479 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.793515 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.794222 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.794869 (+ 647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.794963 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.794985 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.795020 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260928 23:20:48.795468 20425 tablet_service.cc:1467] Processing CreateTablet for tablet 7e84926f916e48b5b78db17fb734c918 (DEFAULT_TABLE table=test-table [id=6bdb7799938b430baf002955fafe10f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:48.795663 20163 tablet_service.cc:1467] Processing CreateTablet for tablet 7e84926f916e48b5b78db17fb734c918 (DEFAULT_TABLE table=test-table [id=6bdb7799938b430baf002955fafe10f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:48.795756 20294 tablet_service.cc:1467] Processing CreateTablet for tablet 7e84926f916e48b5b78db17fb734c918 (DEFAULT_TABLE table=test-table [id=6bdb7799938b430baf002955fafe10f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:20:48.795861 20425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e84926f916e48b5b78db17fb734c918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:48.795962 20163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e84926f916e48b5b78db17fb734c918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:48.796069 20294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e84926f916e48b5b78db17fb734c918. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:48.798393 20505 tablet_bootstrap.cc:492] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: Bootstrap starting.
I20260928 23:20:48.798942 20507 tablet_bootstrap.cc:492] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: Bootstrap starting.
I20260928 23:20:48.799064 20505 tablet_bootstrap.cc:654] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:48.799149 20506 tablet_bootstrap.cc:492] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639: Bootstrap starting.
I20260928 23:20:48.799379 20505 log.cc:824] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:48.799747 20506 tablet_bootstrap.cc:654] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:48.800029 20506 log.cc:824] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:48.800055 20507 tablet_bootstrap.cc:654] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:48.800151 20505 tablet_bootstrap.cc:492] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: No bootstrap required, opened a new log
I20260928 23:20:48.800225 20505 ts_tablet_manager.cc:1397] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:20:48.800323 20507 log.cc:824] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:48.800813 20506 tablet_bootstrap.cc:492] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639: No bootstrap required, opened a new log
I20260928 23:20:48.800930 20507 tablet_bootstrap.cc:492] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: No bootstrap required, opened a new log
I20260928 23:20:48.800935 20506 ts_tablet_manager.cc:1397] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:20:48.800987 20507 ts_tablet_manager.cc:1397] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:20:48.802618 20505 raft_consensus.cc:348] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49d267e306b41d08461e25605e2e639" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37079 } } peers { permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 } } peers { permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 } }
I20260928 23:20:48.802783 20505 raft_consensus.cc:374] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:48.802726 20507 raft_consensus.cc:348] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49d267e306b41d08461e25605e2e639" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37079 } } peers { permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 } } peers { permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 } }
I20260928 23:20:48.802816 20505 raft_consensus.cc:729] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c989c7a1a344da985163d9e9360d58, State: Initialized, Role: FOLLOWER
I20260928 23:20:48.802873 20507 raft_consensus.cc:374] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:48.802930 20507 raft_consensus.cc:729] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a85126635a4982a7a2acb156835b54, State: Initialized, Role: FOLLOWER
I20260928 23:20:48.802953 20505 consensus_queue.cc:260] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49d267e306b41d08461e25605e2e639" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37079 } } peers { permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 } } peers { permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 } }
I20260928 23:20:48.803046 20507 consensus_queue.cc:260] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49d267e306b41d08461e25605e2e639" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37079 } } peers { permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 } } peers { permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 } }
I20260928 23:20:48.803284 20507 ts_tablet_manager.cc:1428] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:20:48.803452 20491 heartbeater.cc:503] Master 127.0.98.62:38849 was elected leader, sending a full tablet report...
I20260928 23:20:48.803449 20505 ts_tablet_manager.cc:1428] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 23:20:48.803610 20360 heartbeater.cc:503] Master 127.0.98.62:38849 was elected leader, sending a full tablet report...
I20260928 23:20:48.803601 20506 raft_consensus.cc:348] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49d267e306b41d08461e25605e2e639" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37079 } } peers { permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 } } peers { permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 } }
I20260928 23:20:48.803743 20506 raft_consensus.cc:374] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:48.803776 20506 raft_consensus.cc:729] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49d267e306b41d08461e25605e2e639, State: Initialized, Role: FOLLOWER
I20260928 23:20:48.803882 20506 consensus_queue.cc:260] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49d267e306b41d08461e25605e2e639" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37079 } } peers { permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 } } peers { permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 } }
I20260928 23:20:48.804103 20506 ts_tablet_manager.cc:1428] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:20:48.804198 20229 heartbeater.cc:503] Master 127.0.98.62:38849 was elected leader, sending a full tablet report...
I20260928 23:20:48.878384 20513 raft_consensus.cc:482] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:20:48.878535 20513 raft_consensus.cc:504] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49d267e306b41d08461e25605e2e639" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37079 } } peers { permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 } } peers { permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 } }
I20260928 23:20:48.878890 20513 leader_election.cc:288] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c989c7a1a344da985163d9e9360d58 (127.0.98.2:33611), d9a85126635a4982a7a2acb156835b54 (127.0.98.3:42727)
I20260928 23:20:48.881629 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.879117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48099 (local address 127.0.98.3:42727)
0928 23:20:48.879162 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.879164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.879173 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:20:48.879248 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.879249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.879258 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.879320 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:48.879480 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.880051 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.880958 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.881319 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.881369 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.881377 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.881511 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.881559 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:20:48.881651 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.879050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42727 (local address 127.0.98.1:48099)
0928 23:20:48.879129 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.879140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.879183 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.879347 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.879348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.879349 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:48.879421 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.879423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.880119 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.880119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.880880 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.880882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.881386 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.881386 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.881451 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.881454 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.881497 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.881555 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 23:20:48.881678 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.879122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53633 (local address 127.0.98.2:33611)
0928 23:20:48.879162 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.879164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.879318 (+ 154us) server_negotiation.cc:407] Connection header received
0928 23:20:48.879383 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.879383 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.879393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.879449 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:20:48.879608 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.880112 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.881016 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.881358 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.881419 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.881428 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.881568 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.881607 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:20:48.881656 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.879055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33611 (local address 127.0.98.1:53633)
0928 23:20:48.879248 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.879258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:20:48.879314 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:20:48.879456 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:20:48.879457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:20:48.879458 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:20:48.879530 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.879532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.880200 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.880201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.880933 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:20:48.880935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.881431 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.881431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:20:48.881487 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.881490 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:20:48.881541 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 23:20:48.881575 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":36,"thread_start_us":99,"threads_started":1}
I20260928 23:20:48.882061 20445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e84926f916e48b5b78db17fb734c918" candidate_uuid: "f49d267e306b41d08461e25605e2e639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a85126635a4982a7a2acb156835b54" is_pre_election: true
I20260928 23:20:48.882131 20314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e84926f916e48b5b78db17fb734c918" candidate_uuid: "f49d267e306b41d08461e25605e2e639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c989c7a1a344da985163d9e9360d58" is_pre_election: true
I20260928 23:20:48.882197 20445 raft_consensus.cc:2454] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49d267e306b41d08461e25605e2e639 in term 0.
I20260928 23:20:48.882270 20314 raft_consensus.cc:2454] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49d267e306b41d08461e25605e2e639 in term 0.
I20260928 23:20:48.882427 20117 leader_election.cc:302] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9a85126635a4982a7a2acb156835b54, f49d267e306b41d08461e25605e2e639; no voters:
I20260928 23:20:48.882560 20513 raft_consensus.cc:2781] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:20:48.882613 20513 raft_consensus.cc:482] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:20:48.882632 20513 raft_consensus.cc:3037] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:48.883335 20513 raft_consensus.cc:504] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49d267e306b41d08461e25605e2e639" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37079 } } peers { permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 } } peers { permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 } }
I20260928 23:20:48.883451 20513 leader_election.cc:288] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [CANDIDATE]: Term 1 election: Requested vote from peers a3c989c7a1a344da985163d9e9360d58 (127.0.98.2:33611), d9a85126635a4982a7a2acb156835b54 (127.0.98.3:42727)
I20260928 23:20:48.883613 20314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e84926f916e48b5b78db17fb734c918" candidate_uuid: "f49d267e306b41d08461e25605e2e639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c989c7a1a344da985163d9e9360d58"
I20260928 23:20:48.883675 20445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e84926f916e48b5b78db17fb734c918" candidate_uuid: "f49d267e306b41d08461e25605e2e639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a85126635a4982a7a2acb156835b54"
I20260928 23:20:48.883714 20314 raft_consensus.cc:3037] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:48.883741 20445 raft_consensus.cc:3037] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:48.884593 20314 raft_consensus.cc:2454] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49d267e306b41d08461e25605e2e639 in term 1.
I20260928 23:20:48.884593 20445 raft_consensus.cc:2454] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49d267e306b41d08461e25605e2e639 in term 1.
I20260928 23:20:48.884763 20116 leader_election.cc:302] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3c989c7a1a344da985163d9e9360d58, f49d267e306b41d08461e25605e2e639; no voters:
I20260928 23:20:48.884845 20513 raft_consensus.cc:2781] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:48.884986 20513 raft_consensus.cc:686] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 1 LEADER]: Becoming Leader. State: Replica: f49d267e306b41d08461e25605e2e639, State: Running, Role: LEADER
I20260928 23:20:48.885072 20513 consensus_queue.cc:237] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49d267e306b41d08461e25605e2e639" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37079 } } peers { permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 } } peers { permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 } }
I20260928 23:20:48.885828 20040 catalog_manager.cc:5495] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 reported cstate change: term changed from 0 to 1, leader changed from <none> to f49d267e306b41d08461e25605e2e639 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f49d267e306b41d08461e25605e2e639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49d267e306b41d08461e25605e2e639" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 } health_report { overall_health: UNKNOWN } } }
W20260928 23:20:48.898851 20361 tablet.cc:2369] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:20:48.911162 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.908963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46880 (local address 127.0.98.1:37079)
0928 23:20:48.909014 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:48.909015 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:48.909023 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:20:48.909080 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:48.909081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:48.909099 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:48.909166 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:20:48.909280 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.909746 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.910472 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:48.910650 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:48.910703 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:48.910845 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:20:48.910942 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:20:48.911012 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:48.911047 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 23:20:48.911066 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:20:48.912925 20314 raft_consensus.cc:1261] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58 [term 1 FOLLOWER]: Refusing update from remote peer f49d267e306b41d08461e25605e2e639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:48.912927 20445 raft_consensus.cc:1261] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54 [term 1 FOLLOWER]: Refusing update from remote peer f49d267e306b41d08461e25605e2e639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:20:48.913247 20515 consensus_queue.cc:1035] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c989c7a1a344da985163d9e9360d58" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:48.913369 20513 consensus_queue.cc:1035] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a85126635a4982a7a2acb156835b54" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:20:48.915508 20518 mvcc.cc:205] Tried to move back new op lower bound from 7334451809944678400 to 7334451809833943040. Current Snapshot: MvccSnapshot[applied={T|T < 7334451809944678400}]
I20260928 23:20:48.915961 20519 mvcc.cc:205] Tried to move back new op lower bound from 7334451809944678400 to 7334451809833943040. Current Snapshot: MvccSnapshot[applied={T|T < 7334451809944678400}]
I20260928 23:20:48.917783 20040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:48.917877 20040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:20:48.918565 20040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1acb4aaa8ebb4af9889ba5728e174ece: Sending DeleteTablet for 3 replicas of tablet 7e84926f916e48b5b78db17fb734c918
I20260928 23:20:48.918867 20294 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e84926f916e48b5b78db17fb734c918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:48 UTC) from {username='slave'} at 127.0.0.1:52066
I20260928 23:20:48.918887 20425 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e84926f916e48b5b78db17fb734c918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:48 UTC) from {username='slave'} at 127.0.0.1:55888
I20260928 23:20:48.918880 20163 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e84926f916e48b5b78db17fb734c918 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:20:48 UTC) from {username='slave'} at 127.0.0.1:46868
I20260928 23:20:48.919087 20530 tablet_replica.cc:330] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639: stopping tablet replica
I20260928 23:20:48.919167 20529 tablet_replica.cc:330] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: stopping tablet replica
I20260928 23:20:48.919183 20530 raft_consensus.cc:2229] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:20:48.919255 20529 raft_consensus.cc:2229] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:48.919265 20530 raft_consensus.cc:2258] T 7e84926f916e48b5b78db17fb734c918 P f49d267e306b41d08461e25605e2e639 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:48.919348 20529 raft_consensus.cc:2258] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:48.919525 20528 tablet_replica.cc:330] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: stopping tablet replica
I20260928 23:20:48.919525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 20101
I20260928 23:20:48.919634 20528 raft_consensus.cc:2229] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:20:48.920082 20528 raft_consensus.cc:2258] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:20:48.920399 20529 ts_tablet_manager.cc:1905] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:48.920440 20528 ts_tablet_manager.cc:1905] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:20:48.922060 20529 ts_tablet_manager.cc:1918] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:48.922106 20528 ts_tablet_manager.cc:1918] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:20:48.922127 20529 log.cc:1196] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-2/wal/wals/7e84926f916e48b5b78db17fb734c918
I20260928 23:20:48.922166 20528 log.cc:1196] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: Deleting WAL directory at /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637417635540-392-0/minicluster-data/ts-1/wal/wals/7e84926f916e48b5b78db17fb734c918
I20260928 23:20:48.922369 20529 ts_tablet_manager.cc:1939] T 7e84926f916e48b5b78db17fb734c918 P d9a85126635a4982a7a2acb156835b54: Deleting consensus metadata
I20260928 23:20:48.922377 20528 ts_tablet_manager.cc:1939] T 7e84926f916e48b5b78db17fb734c918 P a3c989c7a1a344da985163d9e9360d58: Deleting consensus metadata
I20260928 23:20:48.922724 20026 catalog_manager.cc:4841] TS d9a85126635a4982a7a2acb156835b54 (127.0.98.3:42727): tablet 7e84926f916e48b5b78db17fb734c918 (table test-table [id=6bdb7799938b430baf002955fafe10f6]) successfully deleted
I20260928 23:20:48.922741 20025 catalog_manager.cc:4841] TS a3c989c7a1a344da985163d9e9360d58 (127.0.98.2:33611): tablet 7e84926f916e48b5b78db17fb734c918 (table test-table [id=6bdb7799938b430baf002955fafe10f6]) successfully deleted
W20260928 23:20:48.924923 20027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37079 (error 108)
I20260928 23:20:48.925130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 20232
I20260928 23:20:48.925930 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:48.925789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37079 (local address 127.0.0.1:46892)
0928 23:20:48.925848 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 23:20:48.925896 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 23:20:48.926098 20027 catalog_manager.cc:4584] TS f49d267e306b41d08461e25605e2e639 (127.0.98.1:37079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e84926f916e48b5b78db17fb734c918: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37079: connect: Connection refused (error 111)
I20260928 23:20:48.930037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 20363
I20260928 23:20:48.933820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 20010
2026-09-28T23:20:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (693 ms)
[----------] 6 tests from EncryptionPolicyTest (7965 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:48 dist-test-slave-djkr krb5kdc[20534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:48 dist-test-slave-djkr krb5kdc[20534](info): set up 2 sockets
Sep 28 23:20:48 dist-test-slave-djkr krb5kdc[20534](info): commencing operation
krb5kdc: starting...
W20260928 23:20:49.875275 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:49 dist-test-slave-djkr krb5kdc[20534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:20:49.984285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:50.061761 20547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:50.061955 20547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:50.061976 20547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:50.063050 20547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:50.063086 20547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:50.063099 20547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:50.063109 20547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:50.063119 20547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:50.064554 20547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43105
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:50.064744 20547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:50.064916 20547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:50.069373 20553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:50.069362 20552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:50.069485 20547 server_base.cc:1034] running on GCE node
W20260928 23:20:50.069362 20555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:50.069795 20547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:50.070060 20547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:50.071264 20547 hybrid_clock.cc:648] HybridClock initialized: now 1790637650071248 us; error 31 us; skew 500 ppm
Sep 28 23:20:50 dist-test-slave-djkr krb5kdc[20534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:50.077351 20547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:20:50.077788 20547 webserver.cc:460] Webserver started at http://127.0.0.1:37959/ using document root <none> and password file <none>
I20260928 23:20:50.077997 20547 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:50.078044 20547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:50.078171 20547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:50.079182 20547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "8184b6fac5c848db9565c83380ef3501"
format_stamp: "Formatted at 2026-09-28 23:20:50 on dist-test-slave-djkr"
server_key: "a4050ee281c49bbff044387e533fbb84"
server_key_iv: "26f44a6ac2548c4fa763c12a6dfda56d"
server_key_version: "encryptionkey@0"
I20260928 23:20:50.079488 20547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "8184b6fac5c848db9565c83380ef3501"
format_stamp: "Formatted at 2026-09-28 23:20:50 on dist-test-slave-djkr"
server_key: "a4050ee281c49bbff044387e533fbb84"
server_key_iv: "26f44a6ac2548c4fa763c12a6dfda56d"
server_key_version: "encryptionkey@0"
I20260928 23:20:50.080681 20547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:20:50.081323 20562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:50.081454 20547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:20:50.081534 20547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "8184b6fac5c848db9565c83380ef3501"
format_stamp: "Formatted at 2026-09-28 23:20:50 on dist-test-slave-djkr"
server_key: "a4050ee281c49bbff044387e533fbb84"
server_key_iv: "26f44a6ac2548c4fa763c12a6dfda56d"
server_key_version: "encryptionkey@0"
I20260928 23:20:50.081595 20547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:50.094870 20547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:50.096719 20547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:50.096845 20547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:50.100474 20547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43105
I20260928 23:20:50.100504 20614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43105 every 8 connection(s)
I20260928 23:20:50.100812 20547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:50.101600 20615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:50.103760 20615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501: Bootstrap starting.
I20260928 23:20:50.104259 20615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:50.104497 20615 log.cc:824] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:50.105094 20615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501: No bootstrap required, opened a new log
I20260928 23:20:50.106587 20615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8184b6fac5c848db9565c83380ef3501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260928 23:20:50.106727 20615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:50.106750 20615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8184b6fac5c848db9565c83380ef3501, State: Initialized, Role: FOLLOWER
I20260928 23:20:50.106851 20615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8184b6fac5c848db9565c83380ef3501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260928 23:20:50.106896 20615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:50.106922 20615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:50.106950 20615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:50.107398 20615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8184b6fac5c848db9565c83380ef3501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260928 23:20:50.107477 20615 leader_election.cc:302] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8184b6fac5c848db9565c83380ef3501; no voters:
I20260928 23:20:50.107620 20615 leader_election.cc:288] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:50.107669 20618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:50.107815 20618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [term 1 LEADER]: Becoming Leader. State: Replica: 8184b6fac5c848db9565c83380ef3501, State: Running, Role: LEADER
I20260928 23:20:50.107857 20615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:50.107903 20618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8184b6fac5c848db9565c83380ef3501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260928 23:20:50.108366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 20547
I20260928 23:20:50.108479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:50.108647 392 external_mini_cluster.cc:1428] Setting key 8e2f24c8abeeb195da6e1254791591ae
I20260928 23:20:50.109112 20619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8184b6fac5c848db9565c83380ef3501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8184b6fac5c848db9565c83380ef3501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260928 23:20:50.109499 20619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:50.109113 20620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8184b6fac5c848db9565c83380ef3501. Latest consensus state: current_term: 1 leader_uuid: "8184b6fac5c848db9565c83380ef3501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8184b6fac5c848db9565c83380ef3501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260928 23:20:50.109607 20620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501 [sys.catalog]: This master's current role is: LEADER
W20260928 23:20:50.109720 20633 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:20:50.109776 20633 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:20:50.109848 20637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:50.110060 20637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:50.111941 20637 catalog_manager.cc:1347] Generated new cluster ID: d02099b115474024a7acc03a089f92b5
I20260928 23:20:50.111991 20637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:20:50 dist-test-slave-djkr krb5kdc[20534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:20:50.118052 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:50.109214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40005 (local address 127.0.0.1:43105)
0928 23:20:50.109461 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:50.109470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:50.109516 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:20:50.109633 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:50.109640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:50.109696 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:50.109877 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:50.110534 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:50.111833 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:50.112626 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:50.112998 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:50.113092 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:50.115202 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:50.115206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:50.115206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:50.115217 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:50.116675 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:50.117018 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:50.117021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:50.117021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:50.117107 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:50.117301 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:50.117302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:50.117302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:50.117515 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:50.117653 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:50.117775 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 23:20:50.117871 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":175,"threads_started":1}
I20260928 23:20:50.118870 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:20:50.125357 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:50.120099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49399 (local address 127.0.0.1:43105)
0928 23:20:50.120131 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:50.120133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:50.120240 (+ 107us) server_negotiation.cc:407] Connection header received
0928 23:20:50.120291 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:50.120291 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:50.120311 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:50.120353 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:50.120811 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:50.121287 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:50.121916 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:50.122138 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:50.122190 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:50.123159 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:50.123160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:50.123161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:50.123167 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:50.124366 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:50.124694 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:50.124695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:50.124696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:50.124745 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:50.124923 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:50.124924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:50.124924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:50.125026 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:50.125106 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:50.125228 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 23:20:50.125281 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:20:50.143262 20637 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:50.143824 20637 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:50.150250 20637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8184b6fac5c848db9565c83380ef3501: Generated new TSK 0
I20260928 23:20:50.150454 20637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:50.153959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 20547
W20260928 23:20:50.157783 20532 connection.cc:535] client connection to 127.0.0.1:43105 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43105): Connection reset by peer (error 104)
2026-09-28T23:20:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:50 dist-test-slave-djkr krb5kdc[20646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:50 dist-test-slave-djkr krb5kdc[20646](info): set up 2 sockets
Sep 28 23:20:50 dist-test-slave-djkr krb5kdc[20646](info): commencing operation
krb5kdc: starting...
W20260928 23:20:51.091457 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:51 dist-test-slave-djkr krb5kdc[20646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:20:51.200534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:51.277938 20659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:51.278136 20659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:51.278167 20659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:51.279295 20659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:51.279336 20659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:51.279356 20659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:51.279371 20659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:51.279386 20659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:51.280885 20659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:51.281124 20659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:51.281319 20659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:51.285745 20665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:51.285745 20667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:51.285847 20659 server_base.cc:1034] running on GCE node
W20260928 23:20:51.285772 20664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:51.286196 20659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:51.286451 20659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:51.287596 20659 hybrid_clock.cc:648] HybridClock initialized: now 1790637651287579 us; error 32 us; skew 500 ppm
Sep 28 23:20:51 dist-test-slave-djkr krb5kdc[20646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:51.289868 20659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:20:51.290302 20659 webserver.cc:460] Webserver started at http://127.0.0.1:40207/ using document root <none> and password file <none>
I20260928 23:20:51.290494 20659 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:51.290539 20659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:51.290663 20659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:51.291692 20659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "0d2951aa9c884475ad29797e643f0ded"
format_stamp: "Formatted at 2026-09-28 23:20:51 on dist-test-slave-djkr"
server_key: "d32929e96e080e7a62f7afe6d4193fdd"
server_key_iv: "9c1b8b873f17ffa811bbb9cf97f982aa"
server_key_version: "encryptionkey@0"
I20260928 23:20:51.292006 20659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d2951aa9c884475ad29797e643f0ded"
format_stamp: "Formatted at 2026-09-28 23:20:51 on dist-test-slave-djkr"
server_key: "d32929e96e080e7a62f7afe6d4193fdd"
server_key_iv: "9c1b8b873f17ffa811bbb9cf97f982aa"
server_key_version: "encryptionkey@0"
I20260928 23:20:51.293211 20659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:20:51.293840 20674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:51.293994 20659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:20:51.294075 20659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "0d2951aa9c884475ad29797e643f0ded"
format_stamp: "Formatted at 2026-09-28 23:20:51 on dist-test-slave-djkr"
server_key: "d32929e96e080e7a62f7afe6d4193fdd"
server_key_iv: "9c1b8b873f17ffa811bbb9cf97f982aa"
server_key_version: "encryptionkey@0"
I20260928 23:20:51.294131 20659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:51.305091 20659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:51.307109 20659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:51.307245 20659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:51.310990 20659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36531
I20260928 23:20:51.311033 20726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36531 every 8 connection(s)
I20260928 23:20:51.311354 20659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:51.312181 20727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:51.314320 20727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded: Bootstrap starting.
I20260928 23:20:51.314610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 20659
I20260928 23:20:51.314745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:51.314971 392 external_mini_cluster.cc:1428] Setting key f90303c34422245048dd85ccfe3315f7
I20260928 23:20:51.315075 20727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:51.315358 20727 log.cc:824] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:51.316138 20727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded: No bootstrap required, opened a new log
I20260928 23:20:51.318410 20727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2951aa9c884475ad29797e643f0ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260928 23:20:51.318544 20727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:51.318567 20727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2951aa9c884475ad29797e643f0ded, State: Initialized, Role: FOLLOWER
I20260928 23:20:51.318670 20727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2951aa9c884475ad29797e643f0ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260928 23:20:51.318737 20727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:51.318756 20727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:51.318799 20727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:51.319475 20727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2951aa9c884475ad29797e643f0ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260928 23:20:51.319612 20727 leader_election.cc:302] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2951aa9c884475ad29797e643f0ded; no voters:
I20260928 23:20:51.319772 20727 leader_election.cc:288] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:51.319811 20732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:51.320006 20727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:51.320235 20732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [term 1 LEADER]: Becoming Leader. State: Replica: 0d2951aa9c884475ad29797e643f0ded, State: Running, Role: LEADER
I20260928 23:20:51.320364 20732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2951aa9c884475ad29797e643f0ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } }
I20260928 23:20:51.320793 20734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d2951aa9c884475ad29797e643f0ded. Latest consensus state: current_term: 1 leader_uuid: "0d2951aa9c884475ad29797e643f0ded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2951aa9c884475ad29797e643f0ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } } }
I20260928 23:20:51.320775 20733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d2951aa9c884475ad29797e643f0ded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2951aa9c884475ad29797e643f0ded" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36531 } } }
I20260928 23:20:51.320910 20734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:51.320917 20733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:51.321141 20743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:51.321349 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:51.315523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37291 (local address 127.0.0.1:36531)
0928 23:20:51.315694 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:51.315702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:51.315730 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:20:51.316280 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:51.316288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:51.316348 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:51.316490 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:51.316676 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:51.318041 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:51.318922 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:51.319526 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:51.319656 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:51.320345 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:51.320349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:51.320349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:51.320545 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:51.320583 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:51.320598 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:51.320658 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:51.321135 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 23:20:51.321164 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260928 23:20:51.321743 20743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:51.322435 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:20:51.323249 20743 catalog_manager.cc:1347] Generated new cluster ID: a3ee08a047a94a60ba807c5911ae297b
I20260928 23:20:51.323299 20743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:51.327090 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:51.323915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39935 (local address 127.0.0.1:36531)
0928 23:20:51.323946 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:51.323947 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:51.323970 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:20:51.323988 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:51.323989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:51.323997 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:51.324048 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:51.324178 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:51.324909 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:51.325873 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:51.326179 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:51.326247 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:51.326871 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:51.326872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:51.326873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:51.326877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:51.326891 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:51.326896 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:51.326956 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:51.327008 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:20:51.327018 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:20:51.338469 20743 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:51.338963 20743 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:51.341488 20743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d2951aa9c884475ad29797e643f0ded: Generated new TSK 0
I20260928 23:20:51.341656 20743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:51.344879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 20659
W20260928 23:20:51.348690 20644 connection.cc:535] client connection to 127.0.0.1:36531 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36531): Connection reset by peer (error 104)
2026-09-28T23:20:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:51 dist-test-slave-djkr krb5kdc[20758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:51 dist-test-slave-djkr krb5kdc[20758](info): set up 2 sockets
Sep 28 23:20:51 dist-test-slave-djkr krb5kdc[20758](info): commencing operation
krb5kdc: starting...
W20260928 23:20:52.287309 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:52 dist-test-slave-djkr krb5kdc[20758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:20:52.397980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:52.475203 20771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:52.475392 20771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:52.475414 20771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:52.476552 20771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:52.476591 20771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:52.476603 20771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:52.476614 20771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:52.476625 20771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:52.478009 20771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38081
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:52.478216 20771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:52.478412 20771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:52.482857 20779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:52.482852 20776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:52.482951 20777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:52.483050 20771 server_base.cc:1034] running on GCE node
I20260928 23:20:52.483355 20771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:52.483654 20771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:52.484802 20771 hybrid_clock.cc:648] HybridClock initialized: now 1790637652484783 us; error 36 us; skew 500 ppm
Sep 28 23:20:52 dist-test-slave-djkr krb5kdc[20758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:52.487071 20771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:20:52.487428 20771 webserver.cc:460] Webserver started at http://127.0.0.1:42801/ using document root <none> and password file <none>
I20260928 23:20:52.487634 20771 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:52.487676 20771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:52.487763 20771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:52.488703 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "c4c413d5d7a245beb9ca5f27e1e962a4"
format_stamp: "Formatted at 2026-09-28 23:20:52 on dist-test-slave-djkr"
server_key: "6153a7bca37577801ca3e36a3e4b3298"
server_key_iv: "2fcbbf71c6be43f0b7362977181b6d5b"
server_key_version: "encryptionkey@0"
I20260928 23:20:52.488981 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4c413d5d7a245beb9ca5f27e1e962a4"
format_stamp: "Formatted at 2026-09-28 23:20:52 on dist-test-slave-djkr"
server_key: "6153a7bca37577801ca3e36a3e4b3298"
server_key_iv: "2fcbbf71c6be43f0b7362977181b6d5b"
server_key_version: "encryptionkey@0"
I20260928 23:20:52.490146 20771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:20:52.490819 20786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:52.490996 20771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:20:52.491081 20771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "c4c413d5d7a245beb9ca5f27e1e962a4"
format_stamp: "Formatted at 2026-09-28 23:20:52 on dist-test-slave-djkr"
server_key: "6153a7bca37577801ca3e36a3e4b3298"
server_key_iv: "2fcbbf71c6be43f0b7362977181b6d5b"
server_key_version: "encryptionkey@0"
I20260928 23:20:52.491137 20771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:52.494158 20771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:52.495893 20771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:52.495985 20771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:52.499449 20771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38081
I20260928 23:20:52.499495 20838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38081 every 8 connection(s)
I20260928 23:20:52.499788 20771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:52.500684 20839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:52.502351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 20771
I20260928 23:20:52.502465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:52.502637 392 external_mini_cluster.cc:1428] Setting key 4b798d96895f5daa3689c940146118b2
I20260928 23:20:52.502930 20839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4: Bootstrap starting.
I20260928 23:20:52.503502 20839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:52.503826 20839 log.cc:824] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:52.504487 20839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4: No bootstrap required, opened a new log
I20260928 23:20:52.504987 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:52.503091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54333 (local address 127.0.0.1:38081)
0928 23:20:52.503266 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:52.503275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:52.503302 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:20:52.503796 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:52.503803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:52.503829 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:52.503954 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:52.504555 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:52.504559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:52.504559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:52.504564 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:52.504609 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:52.504624 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:52.504696 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:52.504763 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 23:20:52.504802 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260928 23:20:52.505885 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:20:52.506588 20839 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c413d5d7a245beb9ca5f27e1e962a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20260928 23:20:52.506717 20839 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:52.506747 20839 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c413d5d7a245beb9ca5f27e1e962a4, State: Initialized, Role: FOLLOWER
I20260928 23:20:52.506866 20839 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c413d5d7a245beb9ca5f27e1e962a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20260928 23:20:52.506939 20839 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:52.506963 20839 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:52.507007 20839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:52.507687 20839 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c413d5d7a245beb9ca5f27e1e962a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20260928 23:20:52.507779 20839 leader_election.cc:302] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4c413d5d7a245beb9ca5f27e1e962a4; no voters:
I20260928 23:20:52.507941 20839 leader_election.cc:288] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:52.507973 20849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:52.508164 20849 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [term 1 LEADER]: Becoming Leader. State: Replica: c4c413d5d7a245beb9ca5f27e1e962a4, State: Running, Role: LEADER
I20260928 23:20:52.508212 20839 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:52.508216 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:52.507180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45137 (local address 127.0.0.1:38081)
0928 23:20:52.507294 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:52.507296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:52.507308 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:20:52.507356 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:52.507357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:52.507361 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:52.507420 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:52.507895 (+ 475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:52.507896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:52.507897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:52.507900 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:52.507914 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:52.507918 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:52.507980 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:52.508029 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 23:20:52.508050 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 23:20:52.508301 20849 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c413d5d7a245beb9ca5f27e1e962a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20260928 23:20:52.508706 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4c413d5d7a245beb9ca5f27e1e962a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c413d5d7a245beb9ca5f27e1e962a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } } }
I20260928 23:20:52.508790 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:52.509008 20857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:52.509584 20857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:52.509022 20851 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4c413d5d7a245beb9ca5f27e1e962a4. Latest consensus state: current_term: 1 leader_uuid: "c4c413d5d7a245beb9ca5f27e1e962a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c413d5d7a245beb9ca5f27e1e962a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } } }
I20260928 23:20:52.510142 20851 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:52.510972 20857 catalog_manager.cc:1347] Generated new cluster ID: 73717ab8180e4b5d8010c0e5a8995d9f
I20260928 23:20:52.511024 20857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:52.529760 20857 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:52.529938 20857 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:52.539673 20857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4c413d5d7a245beb9ca5f27e1e962a4: Generated new TSK 0
I20260928 23:20:52.539865 20857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:52.547466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 20771
2026-09-28T23:20:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:52 dist-test-slave-djkr krb5kdc[20870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:52 dist-test-slave-djkr krb5kdc[20870](info): set up 2 sockets
Sep 28 23:20:52 dist-test-slave-djkr krb5kdc[20870](info): commencing operation
krb5kdc: starting...
W20260928 23:20:53.491309 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:53 dist-test-slave-djkr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:20:53.601783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:53.679510 20883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:53.679764 20883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:53.679785 20883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:53.680900 20883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:53.680932 20883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:53.680945 20883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:53.680956 20883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:53.680966 20883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:53.682353 20883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:53.682531 20883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:53.682708 20883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:53.687261 20889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:53.687296 20888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:53.687263 20891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:53.687276 20883 server_base.cc:1034] running on GCE node
I20260928 23:20:53.687693 20883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:53.687995 20883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:53.689126 20883 hybrid_clock.cc:648] HybridClock initialized: now 1790637653689133 us; error 44 us; skew 500 ppm
Sep 28 23:20:53 dist-test-slave-djkr krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:53.691304 20883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:20:53.691772 20883 webserver.cc:460] Webserver started at http://127.0.0.1:43559/ using document root <none> and password file <none>
I20260928 23:20:53.691982 20883 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:53.692027 20883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:53.692150 20883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:53.693115 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "ba46cbb95eb547c482c5089ea4e31315"
format_stamp: "Formatted at 2026-09-28 23:20:53 on dist-test-slave-djkr"
server_key: "b3289dfa29c21e7e11a4738531a796f7"
server_key_iv: "ba6689777ad4e5c5df8de8aeb5bb93b7"
server_key_version: "encryptionkey@0"
I20260928 23:20:53.693431 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba46cbb95eb547c482c5089ea4e31315"
format_stamp: "Formatted at 2026-09-28 23:20:53 on dist-test-slave-djkr"
server_key: "b3289dfa29c21e7e11a4738531a796f7"
server_key_iv: "ba6689777ad4e5c5df8de8aeb5bb93b7"
server_key_version: "encryptionkey@0"
I20260928 23:20:53.694608 20883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:20:53.695226 20898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:53.695382 20883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:53.695464 20883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "ba46cbb95eb547c482c5089ea4e31315"
format_stamp: "Formatted at 2026-09-28 23:20:53 on dist-test-slave-djkr"
server_key: "b3289dfa29c21e7e11a4738531a796f7"
server_key_iv: "ba6689777ad4e5c5df8de8aeb5bb93b7"
server_key_version: "encryptionkey@0"
I20260928 23:20:53.695531 20883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:53.705683 20883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:53.707288 20883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:53.707410 20883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:53.711387 20883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35273
I20260928 23:20:53.711436 20950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35273 every 8 connection(s)
I20260928 23:20:53.711889 20883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:53.712764 20951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:53.715644 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315: Bootstrap starting.
I20260928 23:20:53.715766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 20883
I20260928 23:20:53.715888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:53.716065 392 external_mini_cluster.cc:1428] Setting key 9902b7d003e834543b8e59af1b8dbcdd
I20260928 23:20:53.716264 20951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:53.716548 20951 log.cc:824] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:53.717176 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315: No bootstrap required, opened a new log
I20260928 23:20:53.718891 20951 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba46cbb95eb547c482c5089ea4e31315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35273 } }
I20260928 23:20:53.719022 20951 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:53.719053 20951 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba46cbb95eb547c482c5089ea4e31315, State: Initialized, Role: FOLLOWER
I20260928 23:20:53.719177 20951 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba46cbb95eb547c482c5089ea4e31315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35273 } }
I20260928 23:20:53.719249 20951 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:53.719277 20951 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:53.719322 20951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:53.719873 20951 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba46cbb95eb547c482c5089ea4e31315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35273 } }
I20260928 23:20:53.719967 20951 leader_election.cc:302] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba46cbb95eb547c482c5089ea4e31315; no voters:
I20260928 23:20:53.720124 20951 leader_election.cc:288] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:53.720151 20956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:53.720273 20956 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [term 1 LEADER]: Becoming Leader. State: Replica: ba46cbb95eb547c482c5089ea4e31315, State: Running, Role: LEADER
I20260928 23:20:53.720402 20951 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:53.720402 20956 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba46cbb95eb547c482c5089ea4e31315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35273 } }
I20260928 23:20:53.720944 20957 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba46cbb95eb547c482c5089ea4e31315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba46cbb95eb547c482c5089ea4e31315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35273 } } }
I20260928 23:20:53.721042 20957 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:53.720949 20958 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba46cbb95eb547c482c5089ea4e31315. Latest consensus state: current_term: 1 leader_uuid: "ba46cbb95eb547c482c5089ea4e31315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba46cbb95eb547c482c5089ea4e31315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35273 } } }
I20260928 23:20:53.721246 20965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:53.721267 20958 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:53.721742 20965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 23:20:53 dist-test-slave-djkr krb5kdc[20870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:20:53.723378 20965 catalog_manager.cc:1347] Generated new cluster ID: 7a89f58e240e47a695aee40d54799346
I20260928 23:20:53.723433 20965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:53.725853 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:53.716536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33935 (local address 127.0.0.1:35273)
0928 23:20:53.716735 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:53.716743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:53.716766 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:20:53.717186 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:53.717192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:53.717250 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:53.717388 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:53.717980 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:53.718851 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:53.719740 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:53.720191 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:53.720293 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:53.722989 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:53.722993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:53.722994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:53.723005 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:53.724540 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:53.724845 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:53.724849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:53.724849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:53.724936 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:53.725101 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:53.725101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:53.725102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:53.725326 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:53.725453 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:53.725562 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 23:20:53.725701 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260928 23:20:53.726619 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:20:53.729805 20965 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:53.730275 20965 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:53.735915 20965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba46cbb95eb547c482c5089ea4e31315: Generated new TSK 0
I20260928 23:20:53.736083 20965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:53.738230 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:53.727866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50161 (local address 127.0.0.1:35273)
0928 23:20:53.727899 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:53.727900 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:53.731745 (+ 3845us) server_negotiation.cc:407] Connection header received
0928 23:20:53.731805 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:53.731805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:53.731815 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:53.731864 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:53.732476 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:53.733199 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:53.734238 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:53.734542 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:53.734589 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:53.735951 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:53.735952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:53.735953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:53.735958 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:53.737283 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:53.737636 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:53.737636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:53.737637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:53.737693 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:53.737859 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:53.737860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:53.737860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:53.737939 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:53.738001 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:53.738122 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:20:53.738176 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:20:53.740125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 20883
2026-09-28T23:20:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1192 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:53 dist-test-slave-djkr krb5kdc[20982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:53 dist-test-slave-djkr krb5kdc[20982](info): set up 2 sockets
Sep 28 23:20:53 dist-test-slave-djkr krb5kdc[20982](info): commencing operation
krb5kdc: starting...
W20260928 23:20:54.683270 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:54 dist-test-slave-djkr krb5kdc[20982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:20:54.792119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:54.872946 20995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:54.873155 20995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:54.873184 20995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:54.874320 20995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:54.874360 20995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:54.874372 20995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:54.874383 20995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:54.874394 20995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:54.875870 20995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:54.876094 20995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:54.876284 20995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:54.880872 21001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:54.880879 21003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:54.880879 21000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:54.880925 20995 server_base.cc:1034] running on GCE node
I20260928 23:20:54.881292 20995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:54.881572 20995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:54.882718 20995 hybrid_clock.cc:648] HybridClock initialized: now 1790637654882701 us; error 33 us; skew 500 ppm
Sep 28 23:20:54 dist-test-slave-djkr krb5kdc[20982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:54.885074 20995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:20:54.885444 20995 webserver.cc:460] Webserver started at http://127.0.0.1:42873/ using document root <none> and password file <none>
I20260928 23:20:54.885627 20995 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:54.885691 20995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:54.885808 20995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:54.886797 20995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58"
format_stamp: "Formatted at 2026-09-28 23:20:54 on dist-test-slave-djkr"
server_key: "554218b4cdfd1e3fbe4e7314e06f1cad"
server_key_iv: "1d1c18a387148b1ed461bb4e176ae22c"
server_key_version: "encryptionkey@0"
I20260928 23:20:54.887118 20995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58"
format_stamp: "Formatted at 2026-09-28 23:20:54 on dist-test-slave-djkr"
server_key: "554218b4cdfd1e3fbe4e7314e06f1cad"
server_key_iv: "1d1c18a387148b1ed461bb4e176ae22c"
server_key_version: "encryptionkey@0"
I20260928 23:20:54.888475 20995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:20:54.889185 21010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:54.889374 20995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:54.889457 20995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58"
format_stamp: "Formatted at 2026-09-28 23:20:54 on dist-test-slave-djkr"
server_key: "554218b4cdfd1e3fbe4e7314e06f1cad"
server_key_iv: "1d1c18a387148b1ed461bb4e176ae22c"
server_key_version: "encryptionkey@0"
I20260928 23:20:54.889523 20995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:54.917532 20995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:54.919718 20995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:54.919852 20995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:54.923494 20995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43023
I20260928 23:20:54.923563 21062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43023 every 8 connection(s)
I20260928 23:20:54.923879 20995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:54.924759 21063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:54.926322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 20995
I20260928 23:20:54.926440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:54.926615 392 external_mini_cluster.cc:1428] Setting key 7f68329ee7d734159464593eca453687
I20260928 23:20:54.927273 21063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58: Bootstrap starting.
I20260928 23:20:54.927856 21063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:54.928113 21063 log.cc:824] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:54.928704 21063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58: No bootstrap required, opened a new log
I20260928 23:20:54.930382 21063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260928 23:20:54.930521 21063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:54.930554 21063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eeeb876ceb841ccbf9b6a2dfae8ba58, State: Initialized, Role: FOLLOWER
I20260928 23:20:54.930675 21063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260928 23:20:54.930751 21063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:54.930779 21063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:54.930838 21063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:54.931456 21063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260928 23:20:54.931602 21063 leader_election.cc:302] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eeeb876ceb841ccbf9b6a2dfae8ba58; no voters:
I20260928 23:20:54.931736 21066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:54.927106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51541 (local address 127.0.0.1:43023)
0928 23:20:54.927264 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:54.927273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:54.927299 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:20:54.927794 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:54.927801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:54.927854 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:54.927987 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:54.928278 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:54.929210 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:54.929947 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:54.930349 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:54.930441 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:54.931057 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:54.931061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:54.931062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:54.931257 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:54.931296 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:54.931311 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:54.931381 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:54.931430 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 23:20:54.931499 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260928 23:20:54.931761 21063 leader_election.cc:288] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:54.931797 21068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:54.931922 21068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [term 1 LEADER]: Becoming Leader. State: Replica: 0eeeb876ceb841ccbf9b6a2dfae8ba58, State: Running, Role: LEADER
I20260928 23:20:54.931975 21063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:54.932011 21068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } }
I20260928 23:20:54.932380 21069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } } }
I20260928 23:20:54.932453 21069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:54.932695 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:20:54.932637 21070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eeeb876ceb841ccbf9b6a2dfae8ba58. Latest consensus state: current_term: 1 leader_uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeeb876ceb841ccbf9b6a2dfae8ba58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43023 } } }
I20260928 23:20:54.932896 21070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:54.933037 21075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:54.933526 21075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:54.935015 21075 catalog_manager.cc:1347] Generated new cluster ID: 35842180a5084ea0a250dccc37315cc7
I20260928 23:20:54.935060 21075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:54.937337 21066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:54.934243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52873 (local address 127.0.0.1:43023)
0928 23:20:54.934276 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:54.934278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:54.934315 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:20:54.934373 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:54.934373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:54.934382 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:54.934437 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:54.934558 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:54.935124 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:54.936160 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:54.936475 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:54.936532 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:54.937127 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:54.937128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:54.937128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:54.937132 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:54.937144 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:54.937148 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:54.937186 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:54.937249 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 23:20:54.937270 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:20:54.947697 21075 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:54.948446 21075 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:54.952525 21075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eeeb876ceb841ccbf9b6a2dfae8ba58: Generated new TSK 0
I20260928 23:20:54.952688 21075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:54.961150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 20995
2026-09-28T23:20:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:55 dist-test-slave-djkr krb5kdc[21094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:55 dist-test-slave-djkr krb5kdc[21094](info): set up 2 sockets
Sep 28 23:20:55 dist-test-slave-djkr krb5kdc[21094](info): commencing operation
krb5kdc: starting...
W20260928 23:20:55.903333 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:55 dist-test-slave-djkr krb5kdc[21094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:20:56.012434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:56.090094 21107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:56.090281 21107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:56.090301 21107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:56.091403 21107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:56.091442 21107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:56.091455 21107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:56.091465 21107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:56.091478 21107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:56.092942 21107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:56.093159 21107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:56.093362 21107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:56.097905 21112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:56.097853 21115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:56.097867 21113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:56.098058 21107 server_base.cc:1034] running on GCE node
I20260928 23:20:56.098304 21107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:56.098564 21107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:56.099696 21107 hybrid_clock.cc:648] HybridClock initialized: now 1790637656099686 us; error 30 us; skew 500 ppm
Sep 28 23:20:56 dist-test-slave-djkr krb5kdc[21094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:56.106515 21107 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:20:56.106927 21107 webserver.cc:460] Webserver started at http://127.0.0.1:40705/ using document root <none> and password file <none>
I20260928 23:20:56.107141 21107 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:56.107183 21107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:56.107301 21107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:56.108367 21107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "d422de3f65f2444fa1121792a5131646"
format_stamp: "Formatted at 2026-09-28 23:20:56 on dist-test-slave-djkr"
server_key: "6d691022c41b4ff8ed773c3f578183ee"
server_key_iv: "8e437362fc064940e6a904846424765d"
server_key_version: "encryptionkey@0"
I20260928 23:20:56.108681 21107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "d422de3f65f2444fa1121792a5131646"
format_stamp: "Formatted at 2026-09-28 23:20:56 on dist-test-slave-djkr"
server_key: "6d691022c41b4ff8ed773c3f578183ee"
server_key_iv: "8e437362fc064940e6a904846424765d"
server_key_version: "encryptionkey@0"
I20260928 23:20:56.109875 21107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:20:56.110488 21122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:56.110659 21107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:20:56.110744 21107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "d422de3f65f2444fa1121792a5131646"
format_stamp: "Formatted at 2026-09-28 23:20:56 on dist-test-slave-djkr"
server_key: "6d691022c41b4ff8ed773c3f578183ee"
server_key_iv: "8e437362fc064940e6a904846424765d"
server_key_version: "encryptionkey@0"
I20260928 23:20:56.110797 21107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:56.113890 21107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:56.115659 21107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:56.115759 21107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:56.119270 21107 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42471
I20260928 23:20:56.119302 21174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42471 every 8 connection(s)
I20260928 23:20:56.119611 21107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:56.120525 21175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:56.122655 21175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646: Bootstrap starting.
I20260928 23:20:56.123142 21175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:56.123366 21175 log.cc:824] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:56.123948 21175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646: No bootstrap required, opened a new log
I20260928 23:20:56.125479 21175 raft_consensus.cc:348] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422de3f65f2444fa1121792a5131646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } }
I20260928 23:20:56.125623 21175 raft_consensus.cc:374] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:56.125644 21175 raft_consensus.cc:729] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d422de3f65f2444fa1121792a5131646, State: Initialized, Role: FOLLOWER
I20260928 23:20:56.125746 21175 consensus_queue.cc:260] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422de3f65f2444fa1121792a5131646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } }
I20260928 23:20:56.125820 21175 raft_consensus.cc:388] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:56.125847 21175 raft_consensus.cc:482] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:56.125878 21175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:56.126384 21175 raft_consensus.cc:504] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422de3f65f2444fa1121792a5131646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } }
I20260928 23:20:56.126462 21175 leader_election.cc:302] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d422de3f65f2444fa1121792a5131646; no voters:
I20260928 23:20:56.126591 21175 leader_election.cc:288] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:56.126629 21178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:56.126646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 21107
I20260928 23:20:56.126731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:56.126739 21178 raft_consensus.cc:686] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [term 1 LEADER]: Becoming Leader. State: Replica: d422de3f65f2444fa1121792a5131646, State: Running, Role: LEADER
I20260928 23:20:56.126808 21178 consensus_queue.cc:237] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422de3f65f2444fa1121792a5131646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } }
I20260928 23:20:56.126856 21175 sys_catalog.cc:564] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:56.126884 392 external_mini_cluster.cc:1428] Setting key 47433a08ee3165d2c75d16157daba9c4
I20260928 23:20:56.127101 21178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d422de3f65f2444fa1121792a5131646. Latest consensus state: current_term: 1 leader_uuid: "d422de3f65f2444fa1121792a5131646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422de3f65f2444fa1121792a5131646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } } }
I20260928 23:20:56.127239 21178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:56.127386 21179 sys_catalog.cc:455] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d422de3f65f2444fa1121792a5131646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d422de3f65f2444fa1121792a5131646" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42471 } } }
I20260928 23:20:56.127499 21179 sys_catalog.cc:458] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:56.127521 21186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:56.128086 21186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:56.129961 21186 catalog_manager.cc:1347] Generated new cluster ID: ccaa94492367478296b0a513bb15c72d
I20260928 23:20:56.130003 21186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:56.130267 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:56.127437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51149 (local address 127.0.0.1:42471)
0928 23:20:56.127626 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:56.127644 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:56.128957 (+ 1313us) server_negotiation.cc:407] Connection header received
0928 23:20:56.129077 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:56.129084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:56.129114 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:56.129250 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:56.129862 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:56.129865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:56.129865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:56.129869 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:56.129907 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:56.129922 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:56.129988 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:56.130057 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 23:20:56.130096 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260928 23:20:56.131244 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:20:56.133664 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:56.132658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44693 (local address 127.0.0.1:42471)
0928 23:20:56.132689 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:56.132691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:56.132851 (+ 160us) server_negotiation.cc:407] Connection header received
0928 23:20:56.132876 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:56.132877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:56.132882 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:56.132943 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:56.133465 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:56.133466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:56.133466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:56.133470 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:56.133484 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:56.133488 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:56.133544 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:56.133587 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 23:20:56.133599 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:20:56.161609 21186 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:56.161768 21186 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:56.171658 21186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d422de3f65f2444fa1121792a5131646: Generated new TSK 0
I20260928 23:20:56.171828 21186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:56.176658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 21107
2026-09-28T23:20:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:56 dist-test-slave-djkr krb5kdc[21206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:56 dist-test-slave-djkr krb5kdc[21206](info): set up 2 sockets
Sep 28 23:20:56 dist-test-slave-djkr krb5kdc[21206](info): commencing operation
krb5kdc: starting...
W20260928 23:20:57.115305 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:57 dist-test-slave-djkr krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:20:57.226118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:57.304008 21219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:57.304212 21219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:57.304243 21219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:57.305405 21219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:57.305449 21219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:57.305469 21219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:57.305483 21219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:57.305498 21219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:57.306962 21219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:57.307204 21219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:57.307406 21219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:57.311954 21225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:57.311969 21224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:57.311985 21219 server_base.cc:1034] running on GCE node
W20260928 23:20:57.311955 21227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:57.312373 21219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:57.312677 21219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:57.313835 21219 hybrid_clock.cc:648] HybridClock initialized: now 1790637657313819 us; error 28 us; skew 500 ppm
Sep 28 23:20:57 dist-test-slave-djkr krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:57.316251 21219 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:20:57.316664 21219 webserver.cc:460] Webserver started at http://127.0.0.1:44203/ using document root <none> and password file <none>
I20260928 23:20:57.316857 21219 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:57.316903 21219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:57.317023 21219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:57.318043 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "89cb14cb4f824359bc627878a7a8a362"
format_stamp: "Formatted at 2026-09-28 23:20:57 on dist-test-slave-djkr"
server_key: "3778fafb64a49735c06f3407ce3ee291"
server_key_iv: "e332188483e41d124c68e9e504ddf7b7"
server_key_version: "encryptionkey@0"
I20260928 23:20:57.318370 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "89cb14cb4f824359bc627878a7a8a362"
format_stamp: "Formatted at 2026-09-28 23:20:57 on dist-test-slave-djkr"
server_key: "3778fafb64a49735c06f3407ce3ee291"
server_key_iv: "e332188483e41d124c68e9e504ddf7b7"
server_key_version: "encryptionkey@0"
I20260928 23:20:57.319615 21219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:20:57.320294 21234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:57.320509 21219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:20:57.320600 21219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "89cb14cb4f824359bc627878a7a8a362"
format_stamp: "Formatted at 2026-09-28 23:20:57 on dist-test-slave-djkr"
server_key: "3778fafb64a49735c06f3407ce3ee291"
server_key_iv: "e332188483e41d124c68e9e504ddf7b7"
server_key_version: "encryptionkey@0"
I20260928 23:20:57.320669 21219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:57.332546 21219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:57.334208 21219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:57.334337 21219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:57.338552 21219 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38295
I20260928 23:20:57.338595 21286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38295 every 8 connection(s)
I20260928 23:20:57.339071 21219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:57.339687 21287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:57.340333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 21219
I20260928 23:20:57.340444 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:57.340636 392 external_mini_cluster.cc:1428] Setting key 1d52d0d14e8ebd1fea451e2de414c8bb
I20260928 23:20:57.342355 21287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362: Bootstrap starting.
I20260928 23:20:57.343086 21287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:57.343362 21287 log.cc:824] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:57.344053 21287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362: No bootstrap required, opened a new log
I20260928 23:20:57.346251 21287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb14cb4f824359bc627878a7a8a362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260928 23:20:57.346378 21287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:57.346400 21287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cb14cb4f824359bc627878a7a8a362, State: Initialized, Role: FOLLOWER
I20260928 23:20:57.346509 21287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb14cb4f824359bc627878a7a8a362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260928 23:20:57.346570 21287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:57.346590 21287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:57.346633 21287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 23:20:57 dist-test-slave-djkr krb5kdc[21206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:20:57.347111 21287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb14cb4f824359bc627878a7a8a362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260928 23:20:57.347208 21287 leader_election.cc:302] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89cb14cb4f824359bc627878a7a8a362; no voters:
I20260928 23:20:57.347357 21287 leader_election.cc:288] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:57.347384 21292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:57.347487 21292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [term 1 LEADER]: Becoming Leader. State: Replica: 89cb14cb4f824359bc627878a7a8a362, State: Running, Role: LEADER
I20260928 23:20:57.347630 21287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:57.347643 21292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb14cb4f824359bc627878a7a8a362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260928 23:20:57.348205 21293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89cb14cb4f824359bc627878a7a8a362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb14cb4f824359bc627878a7a8a362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } } }
I20260928 23:20:57.348330 21293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:57.348471 21293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89cb14cb4f824359bc627878a7a8a362. Latest consensus state: current_term: 1 leader_uuid: "89cb14cb4f824359bc627878a7a8a362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89cb14cb4f824359bc627878a7a8a362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } } }
I20260928 23:20:57.348526 21293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362 [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:57.348608 21301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:57.349143 21301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:57.350626 21301 catalog_manager.cc:1347] Generated new cluster ID: fc30811d10d24fb2ad3845fb09b988ae
I20260928 23:20:57.350675 21301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:57.351287 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:57.341132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:47905 (local address 127.0.0.1:38295)
0928 23:20:57.341298 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:57.341306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:57.341331 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:20:57.341734 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:57.341740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:57.341790 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:57.341913 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:57.342544 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:57.343864 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:57.344620 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:57.345235 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:57.345351 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:57.347614 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:57.347618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:57.347618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:57.347630 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:57.349651 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:57.350018 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:57.350021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:57.350022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:57.350126 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:57.350364 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:57.350366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:57.350366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:57.350648 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:57.350793 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:57.351028 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 23:20:57.351140 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260928 23:20:57.352208 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:20:57.364406 21301 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:57.365129 21301 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:57.368759 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:57.362994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:56557 (local address 127.0.0.1:38295)
0928 23:20:57.363042 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:57.363044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:57.363155 (+ 111us) server_negotiation.cc:407] Connection header received
0928 23:20:57.363211 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:57.363212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:57.363222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:57.363273 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:57.363790 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:57.364403 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:57.365090 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:57.365364 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:57.365418 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:57.366509 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:57.366510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:57.366510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:20:57.366516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:57.367777 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:57.368090 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:57.368091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:57.368092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:57.368140 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:20:57.368319 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:20:57.368320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:20:57.368320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:20:57.368417 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:57.368517 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:57.368622 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 23:20:57.368677 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:20:57.376086 21301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89cb14cb4f824359bc627878a7a8a362: Generated new TSK 0
I20260928 23:20:57.376260 21301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:57.380717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 21219
W20260928 23:20:57.384371 21204 connection.cc:535] client connection to 127.0.0.1:38295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38295): Connection reset by peer (error 104)
2026-09-28T23:20:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:57 dist-test-slave-djkr krb5kdc[21318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:57 dist-test-slave-djkr krb5kdc[21318](info): set up 2 sockets
Sep 28 23:20:57 dist-test-slave-djkr krb5kdc[21318](info): commencing operation
krb5kdc: starting...
W20260928 23:20:58.323316 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:58 dist-test-slave-djkr krb5kdc[21318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:20:58.430053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:58.506951 21331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:58.507158 21331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:58.507190 21331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:58.508322 21331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:58.508361 21331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:58.508380 21331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:58.508396 21331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:58.508411 21331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:58.509815 21331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:58.510042 21331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:58.510238 21331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:58.514778 21339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:58.514783 21336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:58.514999 21331 server_base.cc:1034] running on GCE node
W20260928 23:20:58.514786 21337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:58.515271 21331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:58.515539 21331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:58.516708 21331 hybrid_clock.cc:648] HybridClock initialized: now 1790637658516695 us; error 28 us; skew 500 ppm
Sep 28 23:20:58 dist-test-slave-djkr krb5kdc[21318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:58.519011 21331 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:20:58.519408 21331 webserver.cc:460] Webserver started at http://127.0.0.1:34499/ using document root <none> and password file <none>
I20260928 23:20:58.519614 21331 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:58.519701 21331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:58.519797 21331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:58.520776 21331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "491906224258464d86e0bb1a677f7ccf"
format_stamp: "Formatted at 2026-09-28 23:20:58 on dist-test-slave-djkr"
server_key: "c38f3e91c586a3dcae4f0be4bb148919"
server_key_iv: "29e2bf17e27ab74541853ac2297f732e"
server_key_version: "encryptionkey@0"
I20260928 23:20:58.521055 21331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "491906224258464d86e0bb1a677f7ccf"
format_stamp: "Formatted at 2026-09-28 23:20:58 on dist-test-slave-djkr"
server_key: "c38f3e91c586a3dcae4f0be4bb148919"
server_key_iv: "29e2bf17e27ab74541853ac2297f732e"
server_key_version: "encryptionkey@0"
I20260928 23:20:58.522186 21331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:20:58.522858 21346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:58.523033 21331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:20:58.523119 21331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "491906224258464d86e0bb1a677f7ccf"
format_stamp: "Formatted at 2026-09-28 23:20:58 on dist-test-slave-djkr"
server_key: "c38f3e91c586a3dcae4f0be4bb148919"
server_key_iv: "29e2bf17e27ab74541853ac2297f732e"
server_key_version: "encryptionkey@0"
I20260928 23:20:58.523180 21331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:58.541580 21331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:58.543334 21331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:58.543448 21331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:58.546959 21331 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35367
I20260928 23:20:58.546983 21398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35367 every 8 connection(s)
I20260928 23:20:58.547299 21331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:58.548096 21399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:58.550211 21399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf: Bootstrap starting.
I20260928 23:20:58.550731 21399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:58.550971 21399 log.cc:824] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:58.551677 21399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf: No bootstrap required, opened a new log
I20260928 23:20:58.553197 21399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491906224258464d86e0bb1a677f7ccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35367 } }
I20260928 23:20:58.553326 21399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:58.553357 21399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491906224258464d86e0bb1a677f7ccf, State: Initialized, Role: FOLLOWER
I20260928 23:20:58.553476 21399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491906224258464d86e0bb1a677f7ccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35367 } }
I20260928 23:20:58.553535 21399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:58.553563 21399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:58.553610 21399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:58.554024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 21331
I20260928 23:20:58.554132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:58.554111 21399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491906224258464d86e0bb1a677f7ccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35367 } }
I20260928 23:20:58.554200 21399 leader_election.cc:302] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 491906224258464d86e0bb1a677f7ccf; no voters:
I20260928 23:20:58.554301 392 external_mini_cluster.cc:1428] Setting key e9a514bbefac89f6846521ce913ea333
I20260928 23:20:58.554343 21399 leader_election.cc:288] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:58.554385 21402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:58.554500 21402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [term 1 LEADER]: Becoming Leader. State: Replica: 491906224258464d86e0bb1a677f7ccf, State: Running, Role: LEADER
I20260928 23:20:58.554621 21399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:58.554634 21402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491906224258464d86e0bb1a677f7ccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35367 } }
I20260928 23:20:58.555058 21403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "491906224258464d86e0bb1a677f7ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491906224258464d86e0bb1a677f7ccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35367 } } }
I20260928 23:20:58.555140 21403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:58.555325 21404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 491906224258464d86e0bb1a677f7ccf. Latest consensus state: current_term: 1 leader_uuid: "491906224258464d86e0bb1a677f7ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "491906224258464d86e0bb1a677f7ccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35367 } } }
I20260928 23:20:58.555430 21404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:58.555649 21408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:58.556183 21408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:58.557526 21408 catalog_manager.cc:1347] Generated new cluster ID: 8405d132a4324b3fb4102726dd444bb1
I20260928 23:20:58.557574 21408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:58.560695 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:58.554822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:36599 (local address 127.0.0.1:35367)
0928 23:20:58.555117 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:58.555126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:58.555155 (+ 29us) server_negotiation.cc:407] Connection header received
0928 23:20:58.555313 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:58.555320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:58.555378 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:58.555818 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:58.555941 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:58.557496 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:58.558592 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:58.559211 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:58.559322 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:58.560030 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:58.560035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:58.560036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:58.560247 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:58.560286 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:58.560301 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:58.560391 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:58.560460 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 23:20:58.560490 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":180,"threads_started":1}
I20260928 23:20:58.561586 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:20:58.566476 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:58.563079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:59941 (local address 127.0.0.1:35367)
0928 23:20:58.563117 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:58.563119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:58.563290 (+ 171us) server_negotiation.cc:407] Connection header received
0928 23:20:58.563311 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:58.563312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:58.563324 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:58.563368 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:58.563506 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:58.564269 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:58.565220 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:20:58.565550 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:20:58.565619 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:20:58.566242 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:58.566243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:58.566243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:58.566247 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:58.566261 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:58.566266 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:58.566325 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:58.566393 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:20:58.566406 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:20:58.567561 21408 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:58.568029 21408 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:58.574465 21408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 491906224258464d86e0bb1a677f7ccf: Generated new TSK 0
I20260928 23:20:58.574608 21408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:58.576876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 21331
W20260928 23:20:58.580803 21316 connection.cc:535] client connection to 127.0.0.1:35367 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35367): Connection reset by peer (error 104)
2026-09-28T23:20:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:58 dist-test-slave-djkr krb5kdc[21430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:58 dist-test-slave-djkr krb5kdc[21430](info): set up 2 sockets
Sep 28 23:20:58 dist-test-slave-djkr krb5kdc[21430](info): commencing operation
krb5kdc: starting...
W20260928 23:20:59.515417 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:20:59 dist-test-slave-djkr krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:20:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:20:59.623236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:20:59.700047 21443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:20:59.700244 21443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:20:59.700265 21443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:20:59.701342 21443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:20:59.701375 21443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:20:59.701386 21443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:20:59.701397 21443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:20:59.701407 21443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:20:59.702781 21443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:20:59.702965 21443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:20:59.703130 21443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:20:59.707568 21451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:59.707731 21448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:20:59.707579 21449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:20:59.707705 21443 server_base.cc:1034] running on GCE node
I20260928 23:20:59.708025 21443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:20:59.708290 21443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:20:59.709501 21443 hybrid_clock.cc:648] HybridClock initialized: now 1790637659709476 us; error 36 us; skew 500 ppm
Sep 28 23:20:59 dist-test-slave-djkr krb5kdc[21430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:20:59.711853 21443 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:20:59.712242 21443 webserver.cc:460] Webserver started at http://127.0.0.1:33353/ using document root <none> and password file <none>
I20260928 23:20:59.712431 21443 fs_manager.cc:359] Metadata directory not provided
I20260928 23:20:59.712477 21443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:20:59.712625 21443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:20:59.713640 21443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "116472151be4488ea5f72ececf5df5bb"
format_stamp: "Formatted at 2026-09-28 23:20:59 on dist-test-slave-djkr"
server_key: "71c03ff4244d92cd2a7eb3b374167a2f"
server_key_iv: "b0dfe9c5a49a988bf8db7fe0ad6d0ea9"
server_key_version: "encryptionkey@0"
I20260928 23:20:59.713963 21443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "116472151be4488ea5f72ececf5df5bb"
format_stamp: "Formatted at 2026-09-28 23:20:59 on dist-test-slave-djkr"
server_key: "71c03ff4244d92cd2a7eb3b374167a2f"
server_key_iv: "b0dfe9c5a49a988bf8db7fe0ad6d0ea9"
server_key_version: "encryptionkey@0"
I20260928 23:20:59.715139 21443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 23:20:59.715813 21458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:59.715941 21443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:20:59.716018 21443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "116472151be4488ea5f72ececf5df5bb"
format_stamp: "Formatted at 2026-09-28 23:20:59 on dist-test-slave-djkr"
server_key: "71c03ff4244d92cd2a7eb3b374167a2f"
server_key_iv: "b0dfe9c5a49a988bf8db7fe0ad6d0ea9"
server_key_version: "encryptionkey@0"
I20260928 23:20:59.716074 21443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:20:59.719115 21443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:20:59.721067 21443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:20:59.721189 21443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:20:59.724750 21443 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37755
I20260928 23:20:59.724781 21510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37755 every 8 connection(s)
I20260928 23:20:59.725057 21443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:20:59.725976 21511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:20:59.726961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 21443
I20260928 23:20:59.727072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:20:59.727262 392 external_mini_cluster.cc:1428] Setting key 5bea15de0e67b8e7005499995e3c5005
I20260928 23:20:59.728605 21511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb: Bootstrap starting.
I20260928 23:20:59.729231 21511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb: Neither blocks nor log segments found. Creating new log.
I20260928 23:20:59.729542 21511 log.cc:824] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb: Log is configured to *not* fsync() on all Append() calls
I20260928 23:20:59.729542 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:59.727772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:57589 (local address 127.0.0.1:37755)
0928 23:20:59.727941 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:59.727949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:59.727973 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:20:59.728419 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:59.728425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:59.728449 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:59.728574 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:59.729194 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:59.729197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:59.729197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:59.729201 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:59.729239 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:59.729248 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:59.729318 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:59.729359 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 23:20:59.729399 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260928 23:20:59.730283 21511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb: No bootstrap required, opened a new log
I20260928 23:20:59.730476 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:20:59.732867 21511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116472151be4488ea5f72ececf5df5bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } }
I20260928 23:20:59.733067 21511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:20:59.733130 21511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116472151be4488ea5f72ececf5df5bb, State: Initialized, Role: FOLLOWER
I20260928 23:20:59.733242 21511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116472151be4488ea5f72ececf5df5bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } }
I20260928 23:20:59.733341 21511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:20:59.733376 21511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:20:59.733453 21511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:20:59.734052 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:20:59.733065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:44327 (local address 127.0.0.1:37755)
0928 23:20:59.733261 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 23:20:59.733262 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:20:59.733272 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:20:59.733316 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:20:59.733317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:20:59.733321 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:20:59.733382 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:20:59.733863 (+ 481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:20:59.733864 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:20:59.733865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:20:59.733869 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:20:59.733884 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:20:59.733888 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:20:59.733948 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:20:59.733980 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 23:20:59.733991 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260928 23:20:59.734134 21511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116472151be4488ea5f72ececf5df5bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } }
I20260928 23:20:59.734228 21511 leader_election.cc:302] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 116472151be4488ea5f72ececf5df5bb; no voters:
I20260928 23:20:59.734395 21511 leader_election.cc:288] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:20:59.734421 21521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:20:59.734560 21521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [term 1 LEADER]: Becoming Leader. State: Replica: 116472151be4488ea5f72ececf5df5bb, State: Running, Role: LEADER
I20260928 23:20:59.734640 21511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:20:59.734649 21521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116472151be4488ea5f72ececf5df5bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } }
I20260928 23:20:59.735039 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "116472151be4488ea5f72ececf5df5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116472151be4488ea5f72ececf5df5bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } } }
I20260928 23:20:59.735050 21522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 116472151be4488ea5f72ececf5df5bb. Latest consensus state: current_term: 1 leader_uuid: "116472151be4488ea5f72ececf5df5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116472151be4488ea5f72ececf5df5bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37755 } } }
I20260928 23:20:59.735199 21522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:59.735389 21530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:20:59.735575 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb [sys.catalog]: This master's current role is: LEADER
I20260928 23:20:59.735915 21530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:20:59.737334 21530 catalog_manager.cc:1347] Generated new cluster ID: 56b342290a66406b9e2dd48bdef77a1c
I20260928 23:20:59.737380 21530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:20:59.743535 21530 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:20:59.743728 21530 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:20:59.748107 21530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 116472151be4488ea5f72ececf5df5bb: Generated new TSK 0
I20260928 23:20:59.748246 21530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:20:59.752487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 21443
2026-09-28T23:20:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1175 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:20:59 dist-test-slave-djkr krb5kdc[21542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:20:59 dist-test-slave-djkr krb5kdc[21542](info): set up 2 sockets
Sep 28 23:20:59 dist-test-slave-djkr krb5kdc[21542](info): commencing operation
krb5kdc: starting...
W20260928 23:21:00.699342 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:00 dist-test-slave-djkr krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:21:00.808444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:00.889185 21555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:00.889377 21555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:00.889407 21555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:00.890529 21555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:00.890618 21555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:00.890652 21555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:00.890667 21555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:00.890681 21555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:00.892174 21555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:00.892364 21555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:00.892537 21555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:00.896946 21561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:00.897053 21563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:00.896955 21560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:00.897186 21555 server_base.cc:1034] running on GCE node
I20260928 23:21:00.897413 21555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:00.897667 21555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:00.898818 21555 hybrid_clock.cc:648] HybridClock initialized: now 1790637660898794 us; error 42 us; skew 500 ppm
Sep 28 23:21:00 dist-test-slave-djkr krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:00.901221 21555 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:21:00.901631 21555 webserver.cc:460] Webserver started at http://127.0.0.1:41871/ using document root <none> and password file <none>
I20260928 23:21:00.901814 21555 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:00.901853 21555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:00.901935 21555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:00.902887 21555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "e4c89ef9ca024341a2032aa0e01b7f79"
format_stamp: "Formatted at 2026-09-28 23:21:00 on dist-test-slave-djkr"
server_key: "a27d83f89f30b67a93631325553d3abb"
server_key_iv: "4474923786776f860e5465c7933a5acb"
server_key_version: "encryptionkey@0"
I20260928 23:21:00.903177 21555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4c89ef9ca024341a2032aa0e01b7f79"
format_stamp: "Formatted at 2026-09-28 23:21:00 on dist-test-slave-djkr"
server_key: "a27d83f89f30b67a93631325553d3abb"
server_key_iv: "4474923786776f860e5465c7933a5acb"
server_key_version: "encryptionkey@0"
I20260928 23:21:00.904369 21555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:21:00.904992 21570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:00.905153 21555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:21:00.905234 21555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "e4c89ef9ca024341a2032aa0e01b7f79"
format_stamp: "Formatted at 2026-09-28 23:21:00 on dist-test-slave-djkr"
server_key: "a27d83f89f30b67a93631325553d3abb"
server_key_iv: "4474923786776f860e5465c7933a5acb"
server_key_version: "encryptionkey@0"
I20260928 23:21:00.905287 21555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:00.915017 21555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:00.916963 21555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:00.917073 21555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:00.920614 21555 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38879
I20260928 23:21:00.920650 21622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38879 every 8 connection(s)
I20260928 23:21:00.920954 21555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:00.921777 21623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:00.922590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 21555
I20260928 23:21:00.922699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:00.922875 392 external_mini_cluster.cc:1428] Setting key 8857a9d2b51a9c50b949390f7f171091
I20260928 23:21:00.924723 21623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79: Bootstrap starting.
I20260928 23:21:00.925266 21623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:00.925539 21623 log.cc:824] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:00.926144 21623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79: No bootstrap required, opened a new log
I20260928 23:21:00.927994 21623 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c89ef9ca024341a2032aa0e01b7f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38879 } }
I20260928 23:21:00.928123 21623 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:21:00.928155 21623 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c89ef9ca024341a2032aa0e01b7f79, State: Initialized, Role: FOLLOWER
I20260928 23:21:00.928277 21623 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c89ef9ca024341a2032aa0e01b7f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38879 } }
I20260928 23:21:00.928356 21623 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:21:00.928388 21623 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:21:00.928444 21623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 23:21:00 dist-test-slave-djkr krb5kdc[21542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:21:00.929064 21623 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c89ef9ca024341a2032aa0e01b7f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38879 } }
I20260928 23:21:00.929159 21623 leader_election.cc:302] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4c89ef9ca024341a2032aa0e01b7f79; no voters:
I20260928 23:21:00.929316 21623 leader_election.cc:288] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:21:00.929343 21628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:00.929452 21628 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [term 1 LEADER]: Becoming Leader. State: Replica: e4c89ef9ca024341a2032aa0e01b7f79, State: Running, Role: LEADER
I20260928 23:21:00.929579 21623 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:00.929586 21628 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c89ef9ca024341a2032aa0e01b7f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38879 } }
I20260928 23:21:00.929984 21630 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4c89ef9ca024341a2032aa0e01b7f79. Latest consensus state: current_term: 1 leader_uuid: "e4c89ef9ca024341a2032aa0e01b7f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c89ef9ca024341a2032aa0e01b7f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38879 } } }
I20260928 23:21:00.929993 21629 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4c89ef9ca024341a2032aa0e01b7f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c89ef9ca024341a2032aa0e01b7f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38879 } } }
I20260928 23:21:00.930140 21630 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:00.930418 21629 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79 [sys.catalog]: This master's current role is: LEADER
W20260928 23:21:00.931248 21643 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:00.931299 21643 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:00.931349 21638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:00.931511 21638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:00.933002 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:00.923362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:49011 (local address 127.0.0.1:38879)
0928 23:21:00.923532 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:00.923540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:00.923630 (+ 90us) server_negotiation.cc:407] Connection header received
0928 23:21:00.924029 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:00.924035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:00.924094 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:00.924216 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:00.924838 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:00.925761 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:00.926534 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:00.926929 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:00.927029 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:00.929345 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:00.929349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:00.929349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:00.929360 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:00.931230 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:00.931686 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:00.931690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:00.931690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:00.931790 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:00.932006 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:00.932007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:00.932007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:00.932292 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:00.932434 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:00.932718 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 23:21:00.932816 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260928 23:21:00.933959 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:00.935889 21638 catalog_manager.cc:1347] Generated new cluster ID: cef323eb0f7c4168859e19a36eb7c0cc
I20260928 23:21:00.935945 21638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:21:00.941243 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:00.935195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:37597 (local address 127.0.0.1:38879)
0928 23:21:00.935227 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:00.935229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:00.935980 (+ 751us) server_negotiation.cc:407] Connection header received
0928 23:21:00.936073 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:00.936074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:00.936084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:00.936129 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:00.936597 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:00.937066 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:00.937723 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:00.937950 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:00.938003 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:00.939025 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:00.939026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:00.939027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:00.939033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:00.940259 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:00.940571 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:00.940572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:00.940572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:00.940621 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:00.940799 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:00.940800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:00.940800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:00.940898 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:00.940981 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:00.941110 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 23:21:00.941164 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:21:00.948269 21638 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:00.948948 21638 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:00.959921 21638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4c89ef9ca024341a2032aa0e01b7f79: Generated new TSK 0
I20260928 23:21:00.960121 21638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:21:00.963104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 21555
W20260928 23:21:00.966984 21540 connection.cc:535] client connection to 127.0.0.1:38879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38879): Connection reset by peer (error 104)
2026-09-28T23:21:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:21:01 dist-test-slave-djkr krb5kdc[21654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:21:01 dist-test-slave-djkr krb5kdc[21654](info): set up 2 sockets
Sep 28 23:21:01 dist-test-slave-djkr krb5kdc[21654](info): commencing operation
krb5kdc: starting...
W20260928 23:21:01.899391 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:01 dist-test-slave-djkr krb5kdc[21654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:21:02.009747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:02.087844 21667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:02.088043 21667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:02.088073 21667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:02.089222 21667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:02.089262 21667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:02.089279 21667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:02.089296 21667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:02.089311 21667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:02.090754 21667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33611
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:02.090996 21667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:02.091192 21667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:02.095734 21675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:02.095749 21673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:02.095779 21672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:02.095939 21667 server_base.cc:1034] running on GCE node
I20260928 23:21:02.096140 21667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:02.096390 21667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:02.097529 21667 hybrid_clock.cc:648] HybridClock initialized: now 1790637662097514 us; error 29 us; skew 500 ppm
Sep 28 23:21:02 dist-test-slave-djkr krb5kdc[21654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:02.103910 21667 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:21:02.104346 21667 webserver.cc:460] Webserver started at http://127.0.0.1:45517/ using document root <none> and password file <none>
I20260928 23:21:02.104571 21667 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:02.104622 21667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:02.104748 21667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:02.105808 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "2dfcafcb857c49428cac55eaf670310b"
format_stamp: "Formatted at 2026-09-28 23:21:02 on dist-test-slave-djkr"
server_key: "560fe70b4c46486bf8775fd8e1ac2559"
server_key_iv: "e98808fe78a7244f31d1f10285bfa758"
server_key_version: "encryptionkey@0"
I20260928 23:21:02.106138 21667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "2dfcafcb857c49428cac55eaf670310b"
format_stamp: "Formatted at 2026-09-28 23:21:02 on dist-test-slave-djkr"
server_key: "560fe70b4c46486bf8775fd8e1ac2559"
server_key_iv: "e98808fe78a7244f31d1f10285bfa758"
server_key_version: "encryptionkey@0"
I20260928 23:21:02.107344 21667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:21:02.108052 21682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:02.108197 21667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:02.108278 21667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "2dfcafcb857c49428cac55eaf670310b"
format_stamp: "Formatted at 2026-09-28 23:21:02 on dist-test-slave-djkr"
server_key: "560fe70b4c46486bf8775fd8e1ac2559"
server_key_iv: "e98808fe78a7244f31d1f10285bfa758"
server_key_version: "encryptionkey@0"
I20260928 23:21:02.108345 21667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:02.117534 21667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:02.119494 21667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:02.119642 21667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:02.123183 21667 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33611
I20260928 23:21:02.123224 21734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33611 every 8 connection(s)
I20260928 23:21:02.123524 21667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:02.123812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 21667
I20260928 23:21:02.123906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:02.124119 392 external_mini_cluster.cc:1428] Setting key 7c25cd21666c6241d25d75f2cb860f73
I20260928 23:21:02.124658 21735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:02.127578 21735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b: Bootstrap starting.
I20260928 23:21:02.128177 21735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:02.128473 21735 log.cc:824] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:02.129122 21735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b: No bootstrap required, opened a new log
I20260928 23:21:02.129444 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:02.124569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:45397 (local address 127.0.0.1:33611)
0928 23:21:02.124849 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:02.124856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:02.124877 (+ 21us) server_negotiation.cc:407] Connection header received
0928 23:21:02.125321 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:02.125328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:02.125376 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:02.125485 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:02.125707 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:02.126565 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:02.127629 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:02.128058 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:02.128140 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:02.128853 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:02.128858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:02.128858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:02.129006 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:02.129034 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:02.129048 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:02.129114 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:02.129217 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 23:21:02.129245 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260928 23:21:02.130357 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:02.131170 21735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfcafcb857c49428cac55eaf670310b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260928 23:21:02.131337 21735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:21:02.131381 21735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfcafcb857c49428cac55eaf670310b, State: Initialized, Role: FOLLOWER
I20260928 23:21:02.131494 21735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfcafcb857c49428cac55eaf670310b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260928 23:21:02.131594 21735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:21:02.131634 21735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:21:02.131721 21735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:02.132313 21735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfcafcb857c49428cac55eaf670310b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260928 23:21:02.132398 21735 leader_election.cc:302] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dfcafcb857c49428cac55eaf670310b; no voters:
I20260928 23:21:02.132570 21735 leader_election.cc:288] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:21:02.132609 21745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:02.132766 21745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [term 1 LEADER]: Becoming Leader. State: Replica: 2dfcafcb857c49428cac55eaf670310b, State: Running, Role: LEADER
I20260928 23:21:02.132815 21735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:02.132874 21745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfcafcb857c49428cac55eaf670310b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } }
I20260928 23:21:02.133195 21745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dfcafcb857c49428cac55eaf670310b. Latest consensus state: current_term: 1 leader_uuid: "2dfcafcb857c49428cac55eaf670310b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfcafcb857c49428cac55eaf670310b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } } }
I20260928 23:21:02.133203 21747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dfcafcb857c49428cac55eaf670310b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfcafcb857c49428cac55eaf670310b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33611 } } }
I20260928 23:21:02.133308 21745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:02.133322 21747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:02.134320 21752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:02.134639 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:02.131589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:49785 (local address 127.0.0.1:33611)
0928 23:21:02.131677 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:02.131678 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:02.131757 (+ 79us) server_negotiation.cc:407] Connection header received
0928 23:21:02.131807 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:02.131808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:02.131817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:02.131864 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:02.132003 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:02.132479 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:02.133415 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:02.133632 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:02.133684 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:02.134290 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:02.134291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:02.134292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:02.134295 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:02.134310 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:02.134314 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:02.134361 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:02.134562 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 23:21:02.134573 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 23:21:02.134852 21752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:02.136130 21752 catalog_manager.cc:1347] Generated new cluster ID: 9706aae56ca040c3b66413588b0efc3d
I20260928 23:21:02.136169 21752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:21:02.153609 21752 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:02.154222 21752 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:02.166320 21752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dfcafcb857c49428cac55eaf670310b: Generated new TSK 0
I20260928 23:21:02.166499 21752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:21:02.172525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 21667
W20260928 23:21:02.176126 21652 connection.cc:535] client connection to 127.0.0.1:33611 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33611): Connection reset by peer (error 104)
2026-09-28T23:21:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:21:02 dist-test-slave-djkr krb5kdc[21766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:21:02 dist-test-slave-djkr krb5kdc[21766](info): set up 2 sockets
Sep 28 23:21:02 dist-test-slave-djkr krb5kdc[21766](info): commencing operation
krb5kdc: starting...
W20260928 23:21:03.131297 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:03 dist-test-slave-djkr krb5kdc[21766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:21:03.240144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:03.317082 21779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:03.317276 21779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:03.317296 21779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:03.318397 21779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:03.318439 21779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:03.318451 21779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:03.318473 21779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:03.318495 21779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:03.319964 21779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:03.320178 21779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:03.320366 21779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:03.324753 21787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:03.324761 21785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:03.324949 21779 server_base.cc:1034] running on GCE node
W20260928 23:21:03.324769 21784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:03.325234 21779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:03.325493 21779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:03.326640 21779 hybrid_clock.cc:648] HybridClock initialized: now 1790637663326629 us; error 25 us; skew 500 ppm
Sep 28 23:21:03 dist-test-slave-djkr krb5kdc[21766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:03.328908 21779 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:21:03.329308 21779 webserver.cc:460] Webserver started at http://127.0.0.1:45367/ using document root <none> and password file <none>
I20260928 23:21:03.329487 21779 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:03.329555 21779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:03.329674 21779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:03.330641 21779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "2ff2d4d751ae4effbcb81d6921978fee"
format_stamp: "Formatted at 2026-09-28 23:21:03 on dist-test-slave-djkr"
server_key: "5fc48d9e99ebc457a3e2a793220712c7"
server_key_iv: "dcc579b011af6016dd41a0b01d6672da"
server_key_version: "encryptionkey@0"
I20260928 23:21:03.330945 21779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ff2d4d751ae4effbcb81d6921978fee"
format_stamp: "Formatted at 2026-09-28 23:21:03 on dist-test-slave-djkr"
server_key: "5fc48d9e99ebc457a3e2a793220712c7"
server_key_iv: "dcc579b011af6016dd41a0b01d6672da"
server_key_version: "encryptionkey@0"
I20260928 23:21:03.332157 21779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:21:03.332793 21794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:03.332931 21779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:21:03.333002 21779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "2ff2d4d751ae4effbcb81d6921978fee"
format_stamp: "Formatted at 2026-09-28 23:21:03 on dist-test-slave-djkr"
server_key: "5fc48d9e99ebc457a3e2a793220712c7"
server_key_iv: "dcc579b011af6016dd41a0b01d6672da"
server_key_version: "encryptionkey@0"
I20260928 23:21:03.333063 21779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:03.336107 21779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:03.337684 21779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:03.337792 21779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:03.341336 21779 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34243
I20260928 23:21:03.341367 21846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34243 every 8 connection(s)
I20260928 23:21:03.341640 21779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:03.342460 21847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:03.343956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 21779
I20260928 23:21:03.344075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:03.344245 392 external_mini_cluster.cc:1428] Setting key 75eea7b4b3c1ee7d89c88db9082d38ed
I20260928 23:21:03.344888 21847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee: Bootstrap starting.
I20260928 23:21:03.345435 21847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:03.345741 21847 log.cc:824] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:03.346449 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:03.344712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:51555 (local address 127.0.0.1:34243)
0928 23:21:03.344860 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:03.344867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:03.344889 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:21:03.345303 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:03.345310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:03.345336 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:03.345445 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:03.346073 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:03.346076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:03.346076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:03.346081 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:03.346112 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:03.346123 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:03.346187 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:03.346243 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:21:03.346283 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260928 23:21:03.346447 21847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee: No bootstrap required, opened a new log
I20260928 23:21:03.347303 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:03.348244 21847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff2d4d751ae4effbcb81d6921978fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260928 23:21:03.348369 21847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:21:03.348389 21847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff2d4d751ae4effbcb81d6921978fee, State: Initialized, Role: FOLLOWER
I20260928 23:21:03.348488 21847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff2d4d751ae4effbcb81d6921978fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260928 23:21:03.348565 21847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:21:03.348596 21847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:21:03.348642 21847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:03.349248 21847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff2d4d751ae4effbcb81d6921978fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260928 23:21:03.349357 21847 leader_election.cc:302] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ff2d4d751ae4effbcb81d6921978fee; no voters:
I20260928 23:21:03.349509 21847 leader_election.cc:288] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:21:03.349555 21857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:03.349623 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:03.348574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.89:47613 (local address 127.0.0.1:34243)
0928 23:21:03.348605 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:03.348607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:03.348802 (+ 195us) server_negotiation.cc:407] Connection header received
0928 23:21:03.348866 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:03.348866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:03.348869 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:03.348912 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:03.349366 (+ 454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:03.349367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:03.349367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:03.349371 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:03.349384 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:03.349388 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:03.349438 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:03.349485 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 23:21:03.349497 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:21:03.349700 21857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [term 1 LEADER]: Becoming Leader. State: Replica: 2ff2d4d751ae4effbcb81d6921978fee, State: Running, Role: LEADER
I20260928 23:21:03.349773 21857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff2d4d751ae4effbcb81d6921978fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260928 23:21:03.349799 21847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:03.350071 21858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ff2d4d751ae4effbcb81d6921978fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff2d4d751ae4effbcb81d6921978fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } } }
I20260928 23:21:03.350081 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ff2d4d751ae4effbcb81d6921978fee. Latest consensus state: current_term: 1 leader_uuid: "2ff2d4d751ae4effbcb81d6921978fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff2d4d751ae4effbcb81d6921978fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } } }
I20260928 23:21:03.350200 21858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:03.350283 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:03.350548 21864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:03.351027 21864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:03.352350 21864 catalog_manager.cc:1347] Generated new cluster ID: ccd024c861004f6fa075d3e39d4ca2bf
I20260928 23:21:03.352408 21864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:21:03.360842 21864 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:03.360991 21864 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:03.366000 21864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ff2d4d751ae4effbcb81d6921978fee: Generated new TSK 0
I20260928 23:21:03.366155 21864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:21:03.374101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 21779
2026-09-28T23:21:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:21:03 dist-test-slave-djkr krb5kdc[21878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:21:03 dist-test-slave-djkr krb5kdc[21878](info): set up 2 sockets
Sep 28 23:21:03 dist-test-slave-djkr krb5kdc[21878](info): commencing operation
krb5kdc: starting...
W20260928 23:21:04.323661 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:04 dist-test-slave-djkr krb5kdc[21878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:21:04.434509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43861
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:04.513198 21891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:04.513394 21891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:04.513433 21891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:04.514546 21891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:04.514585 21891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:04.514603 21891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:04.514618 21891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:04.514633 21891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:04.516126 21891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:04.516356 21891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:04.516562 21891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:04.521121 21897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:04.521150 21896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:04.521099 21899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:04.521365 21891 server_base.cc:1034] running on GCE node
I20260928 23:21:04.521579 21891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:04.521818 21891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:04.522949 21891 hybrid_clock.cc:648] HybridClock initialized: now 1790637664522936 us; error 29 us; skew 500 ppm
Sep 28 23:21:04 dist-test-slave-djkr krb5kdc[21878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:04.525445 21891 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:21:04.525887 21891 webserver.cc:460] Webserver started at http://127.0.98.62:40277/ using document root <none> and password file <none>
I20260928 23:21:04.526094 21891 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:04.526158 21891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:04.526281 21891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:04.527287 21891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "ca6c173dde4144feb26077419d5352af"
format_stamp: "Formatted at 2026-09-28 23:21:04 on dist-test-slave-djkr"
server_key: "b9d29ba9df1994e6630182f6c0f81e94"
server_key_iv: "6e733fc265792f80ac311b2cb1cc924b"
server_key_version: "encryptionkey@0"
I20260928 23:21:04.527652 21891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca6c173dde4144feb26077419d5352af"
format_stamp: "Formatted at 2026-09-28 23:21:04 on dist-test-slave-djkr"
server_key: "b9d29ba9df1994e6630182f6c0f81e94"
server_key_iv: "6e733fc265792f80ac311b2cb1cc924b"
server_key_version: "encryptionkey@0"
I20260928 23:21:04.528836 21891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:21:04.529524 21906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:04.529665 21891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:04.529744 21891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "ca6c173dde4144feb26077419d5352af"
format_stamp: "Formatted at 2026-09-28 23:21:04 on dist-test-slave-djkr"
server_key: "b9d29ba9df1994e6630182f6c0f81e94"
server_key_iv: "6e733fc265792f80ac311b2cb1cc924b"
server_key_version: "encryptionkey@0"
I20260928 23:21:04.529808 21891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:04.544587 21891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:04.546270 21891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:04.546398 21891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:04.550238 21891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43861
I20260928 23:21:04.550275 21958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43861 every 8 connection(s)
I20260928 23:21:04.550585 21891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:04.551359 21959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:04.553550 21959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af: Bootstrap starting.
I20260928 23:21:04.554086 21959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:04.554329 21959 log.cc:824] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:04.554962 21959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af: No bootstrap required, opened a new log
I20260928 23:21:04.556849 21959 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6c173dde4144feb26077419d5352af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260928 23:21:04.556981 21959 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:21:04.557004 21959 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6c173dde4144feb26077419d5352af, State: Initialized, Role: FOLLOWER
I20260928 23:21:04.557125 21959 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6c173dde4144feb26077419d5352af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260928 23:21:04.557186 21959 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:21:04.557210 21959 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:21:04.557255 21959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:04.557744 21959 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6c173dde4144feb26077419d5352af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260928 23:21:04.557825 21959 leader_election.cc:302] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca6c173dde4144feb26077419d5352af; no voters:
I20260928 23:21:04.557960 21959 leader_election.cc:288] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:21:04.557999 21962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:04.558183 21962 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [term 1 LEADER]: Becoming Leader. State: Replica: ca6c173dde4144feb26077419d5352af, State: Running, Role: LEADER
I20260928 23:21:04.558182 21959 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:04.558328 21962 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6c173dde4144feb26077419d5352af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260928 23:21:04.558823 21964 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6c173dde4144feb26077419d5352af. Latest consensus state: current_term: 1 leader_uuid: "ca6c173dde4144feb26077419d5352af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6c173dde4144feb26077419d5352af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } }
I20260928 23:21:04.559108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 21891
I20260928 23:21:04.559193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:04.559384 392 external_mini_cluster.cc:1428] Setting key 93f8b183f533becc492ba8dcead234be
W20260928 23:21:04.559984 21977 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:04.560056 21977 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:04.559796 21963 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca6c173dde4144feb26077419d5352af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6c173dde4144feb26077419d5352af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } }
I20260928 23:21:04.560153 21963 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:04.558910 21964 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:04.560348 21981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:04.560505 21981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:04.561975 21981 catalog_manager.cc:1347] Generated new cluster ID: 98b34e647d944f5c97e1a6892c3dae54
I20260928 23:21:04.562024 21981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:21:04 dist-test-slave-djkr krb5kdc[21878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:21:04.570819 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:04.560012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60463 (local address 127.0.98.62:43861)
0928 23:21:04.560160 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:04.560169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:04.560463 (+ 294us) server_negotiation.cc:407] Connection header received
0928 23:21:04.560619 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:04.560625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:04.560667 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:04.560775 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:04.561410 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:04.562752 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:04.563807 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:04.564452 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:04.564564 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:04.567237 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:04.567242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:04.567242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:04.567253 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:04.569092 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:04.569553 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:04.569555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:04.569556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:04.569652 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:04.569858 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:04.569858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:04.569859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:04.570147 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:04.570288 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:04.570507 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 23:21:04.570600 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260928 23:21:04.571777 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:04.575294 21981 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:04.575860 21981 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:04.580433 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:04.573263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50919 (local address 127.0.98.62:43861)
0928 23:21:04.573295 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:04.573297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:04.573305 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:21:04.573327 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:04.573328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:04.573337 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:04.573397 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:04.573952 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:04.574735 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:04.575538 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:04.575885 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:04.575950 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:04.577427 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:04.577428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:04.577429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:04.577435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:04.578973 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:04.579392 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:04.579393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:04.579393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:04.579446 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:04.579689 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:04.579690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:04.579690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:04.579809 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:04.579889 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:04.580279 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 23:21:04.580343 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:21:04.585057 21981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca6c173dde4144feb26077419d5352af: Generated new TSK 0
I20260928 23:21:04.585204 21981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:21:04.588215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 21891
W20260928 23:21:04.591985 21876 connection.cc:535] client connection to 127.0.98.62:43861 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43861): Connection reset by peer (error 104)
2026-09-28T23:21:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:21:04 dist-test-slave-djkr krb5kdc[21990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:21:04 dist-test-slave-djkr krb5kdc[21990](info): set up 2 sockets
Sep 28 23:21:04 dist-test-slave-djkr krb5kdc[21990](info): commencing operation
krb5kdc: starting...
W20260928 23:21:05.523295 392 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:05 dist-test-slave-djkr krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:21:05.632517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46839
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:05.709105 22003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:05.709306 22003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:05.709334 22003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:05.710469 22003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:05.710510 22003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:05.710529 22003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:05.710544 22003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:05.710561 22003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:05.712025 22003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:05.712265 22003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:05.712469 22003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:05.717067 22011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:05.717155 22008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:05.717156 22003 server_base.cc:1034] running on GCE node
W20260928 23:21:05.717170 22009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:05.717509 22003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:05.717772 22003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:05.718922 22003 hybrid_clock.cc:648] HybridClock initialized: now 1790637665718904 us; error 33 us; skew 500 ppm
Sep 28 23:21:05 dist-test-slave-djkr krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:05.721213 22003 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:21:05.721669 22003 webserver.cc:460] Webserver started at http://127.0.98.62:38051/ using document root <none> and password file <none>
I20260928 23:21:05.721877 22003 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:05.721923 22003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:05.722045 22003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:05.723038 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "84a41431cd864381b5791ae3afcd7d68"
format_stamp: "Formatted at 2026-09-28 23:21:05 on dist-test-slave-djkr"
server_key: "07ef357bf3fef41fe2565cea060f58bd"
server_key_iv: "ef1ca89a971b8b7da3a8e95662711dab"
server_key_version: "encryptionkey@0"
I20260928 23:21:05.723347 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "84a41431cd864381b5791ae3afcd7d68"
format_stamp: "Formatted at 2026-09-28 23:21:05 on dist-test-slave-djkr"
server_key: "07ef357bf3fef41fe2565cea060f58bd"
server_key_iv: "ef1ca89a971b8b7da3a8e95662711dab"
server_key_version: "encryptionkey@0"
I20260928 23:21:05.724659 22003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:21:05.725368 22018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:05.725522 22003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:21:05.725627 22003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "84a41431cd864381b5791ae3afcd7d68"
format_stamp: "Formatted at 2026-09-28 23:21:05 on dist-test-slave-djkr"
server_key: "07ef357bf3fef41fe2565cea060f58bd"
server_key_iv: "ef1ca89a971b8b7da3a8e95662711dab"
server_key_version: "encryptionkey@0"
I20260928 23:21:05.725690 22003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:05.736088 22003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:05.737959 22003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:05.738085 22003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:05.741739 22003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46839
I20260928 23:21:05.741771 22070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46839 every 8 connection(s)
I20260928 23:21:05.742079 22003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:05.742877 22071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:05.745102 22071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68: Bootstrap starting.
I20260928 23:21:05.745594 22071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:05.745805 22071 log.cc:824] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:05.746381 22071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68: No bootstrap required, opened a new log
I20260928 23:21:05.746598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 22003
I20260928 23:21:05.746682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:05.746810 392 external_mini_cluster.cc:1428] Setting key 2dc51f51d9d4de35c87c76c02c257297
I20260928 23:21:05.748158 22071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a41431cd864381b5791ae3afcd7d68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } }
I20260928 23:21:05.748299 22071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:21:05.748332 22071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a41431cd864381b5791ae3afcd7d68, State: Initialized, Role: FOLLOWER
I20260928 23:21:05.748512 22071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a41431cd864381b5791ae3afcd7d68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } }
I20260928 23:21:05.748594 22071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:21:05.748616 22071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:21:05.748660 22071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:05.749158 22071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a41431cd864381b5791ae3afcd7d68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } }
I20260928 23:21:05.749238 22071 leader_election.cc:302] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84a41431cd864381b5791ae3afcd7d68; no voters:
I20260928 23:21:05.749384 22071 leader_election.cc:288] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:21:05.749423 22076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:05.749577 22076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [term 1 LEADER]: Becoming Leader. State: Replica: 84a41431cd864381b5791ae3afcd7d68, State: Running, Role: LEADER
I20260928 23:21:05.749620 22071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:05.749660 22076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a41431cd864381b5791ae3afcd7d68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } }
I20260928 23:21:05.749974 22076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84a41431cd864381b5791ae3afcd7d68. Latest consensus state: current_term: 1 leader_uuid: "84a41431cd864381b5791ae3afcd7d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a41431cd864381b5791ae3afcd7d68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } } }
I20260928 23:21:05.750190 22076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:05.750126 22077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84a41431cd864381b5791ae3afcd7d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84a41431cd864381b5791ae3afcd7d68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46839 } } }
I20260928 23:21:05.750568 22077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68 [sys.catalog]: This master's current role is: LEADER
W20260928 23:21:05.751497 22092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:05.751612 22092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:05.751798 22085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:05.752014 22085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:05.753345 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:05.747237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54033 (local address 127.0.98.62:46839)
0928 23:21:05.747408 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:05.747415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:05.747439 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:21:05.748131 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:05.748137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:05.748183 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:05.748290 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:05.748544 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:05.749969 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:05.751107 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:05.751690 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:05.752096 (+ 406us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:05.752487 (+ 391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:05.752491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:05.752492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:05.752721 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:05.752760 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:05.752775 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:05.753054 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:05.753095 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 23:21:05.753124 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260928 23:21:05.754441 22085 catalog_manager.cc:1347] Generated new cluster ID: fb6cf3070daf40ff8f0b1cced6411781
I20260928 23:21:05.754495 22085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:21:05.754528 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:05.758342 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:05.755762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34089 (local address 127.0.98.62:46839)
0928 23:21:05.755799 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:05.755800 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:05.755914 (+ 114us) server_negotiation.cc:407] Connection header received
0928 23:21:05.755968 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:05.755968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:05.755978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:05.756028 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:05.756162 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:05.756622 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:05.757292 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:05.757507 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:05.757558 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:05.758136 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:05.758137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:05.758138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:05.758142 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:05.758154 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:05.758158 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:05.758207 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:05.758258 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 23:21:05.758269 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:21:05.771035 22085 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:05.771514 22085 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:05.785142 22085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84a41431cd864381b5791ae3afcd7d68: Generated new TSK 0
I20260928 23:21:05.785329 22085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:21:05.791638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 22003
W20260928 23:21:05.795207 21988 connection.cc:535] client connection to 127.0.98.62:46839 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46839): Connection reset by peer (error 104)
2026-09-28T23:21:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:21:05 dist-test-slave-djkr krb5kdc[22102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:21:05 dist-test-slave-djkr krb5kdc[22102](info): set up 2 sockets
Sep 28 23:21:05 dist-test-slave-djkr krb5kdc[22102](info): commencing operation
krb5kdc: starting...
W20260928 23:21:06.739400 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:06 dist-test-slave-djkr krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:21:06.850174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:06.928790 22115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:06.928989 22115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:06.929020 22115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:06.930133 22115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:06.930174 22115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:06.930192 22115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:06.930208 22115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:06.930224 22115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:06.931716 22115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40523
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:06.931945 22115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:06.932140 22115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:06.936760 22121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:06.936755 22120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:06.936756 22123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:06.936833 22115 server_base.cc:1034] running on GCE node
I20260928 23:21:06.937230 22115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:06.937525 22115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:06.938676 22115 hybrid_clock.cc:648] HybridClock initialized: now 1790637666938661 us; error 27 us; skew 500 ppm
Sep 28 23:21:06 dist-test-slave-djkr krb5kdc[22102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:06.941123 22115 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:21:06.941509 22115 webserver.cc:460] Webserver started at http://127.0.98.62:40981/ using document root <none> and password file <none>
I20260928 23:21:06.941711 22115 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:06.941757 22115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:06.941874 22115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:06.942869 22115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd"
format_stamp: "Formatted at 2026-09-28 23:21:06 on dist-test-slave-djkr"
server_key: "614964384610372cceef43a7360f7f80"
server_key_iv: "be45954c42c0b9bc7fd8d849840cf0a4"
server_key_version: "encryptionkey@0"
I20260928 23:21:06.943187 22115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd"
format_stamp: "Formatted at 2026-09-28 23:21:06 on dist-test-slave-djkr"
server_key: "614964384610372cceef43a7360f7f80"
server_key_iv: "be45954c42c0b9bc7fd8d849840cf0a4"
server_key_version: "encryptionkey@0"
I20260928 23:21:06.944450 22115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:21:06.945108 22130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:06.945276 22115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:21:06.945366 22115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd"
format_stamp: "Formatted at 2026-09-28 23:21:06 on dist-test-slave-djkr"
server_key: "614964384610372cceef43a7360f7f80"
server_key_iv: "be45954c42c0b9bc7fd8d849840cf0a4"
server_key_version: "encryptionkey@0"
I20260928 23:21:06.945429 22115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:06.948608 22115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:06.950271 22115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:06.950390 22115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:06.953977 22115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40523
I20260928 23:21:06.954042 22182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40523 every 8 connection(s)
I20260928 23:21:06.954301 22115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:06.955230 22183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:06.957388 22183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd: Bootstrap starting.
I20260928 23:21:06.957904 22183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:06.958150 22183 log.cc:824] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:06.958845 22183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd: No bootstrap required, opened a new log
I20260928 23:21:06.960412 22183 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } }
I20260928 23:21:06.960564 22183 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:21:06.960587 22183 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb44b4c45e3a44e1a6fd49e7816ee8cd, State: Initialized, Role: FOLLOWER
I20260928 23:21:06.960711 22183 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } }
I20260928 23:21:06.960778 22183 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:21:06.960810 22183 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:21:06.960855 22183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:06.961337 22183 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } }
I20260928 23:21:06.961421 22183 leader_election.cc:302] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb44b4c45e3a44e1a6fd49e7816ee8cd; no voters:
I20260928 23:21:06.961572 22183 leader_election.cc:288] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:21:06.961607 22186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:06.961759 22186 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [term 1 LEADER]: Becoming Leader. State: Replica: fb44b4c45e3a44e1a6fd49e7816ee8cd, State: Running, Role: LEADER
I20260928 23:21:06.961797 22183 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:06.961843 22186 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } }
I20260928 23:21:06.962177 22188 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb44b4c45e3a44e1a6fd49e7816ee8cd. Latest consensus state: current_term: 1 leader_uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } } }
I20260928 23:21:06.962173 22187 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb44b4c45e3a44e1a6fd49e7816ee8cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40523 } } }
I20260928 23:21:06.962292 22188 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:06.962363 22187 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:06.962736 22191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:06.963258 22191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:06.964219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 22115
I20260928 23:21:06.964331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:06.964525 392 external_mini_cluster.cc:1428] Setting key 4b634e126c3a1d06e4c5698d1c2555aa
I20260928 23:21:06.966394 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:06.964923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34217 (local address 127.0.98.62:40523)
0928 23:21:06.965069 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:06.965076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:06.965171 (+ 95us) server_negotiation.cc:407] Connection header received
0928 23:21:06.965310 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:06.965315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:06.965337 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:06.965439 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:06.966022 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:06.966024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:06.966025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:06.966028 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:06.966063 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:06.966073 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:06.966119 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:06.966194 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 23:21:06.966225 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260928 23:21:06.967206 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:06.967883 22191 catalog_manager.cc:1347] Generated new cluster ID: 8618c314b664455d93b5b0d35a8c32e5
I20260928 23:21:06.967931 22191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:21:06.972419 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:06.968709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42425 (local address 127.0.98.62:40523)
0928 23:21:06.971601 (+ 2892us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:06.971603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:06.971610 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:21:06.971628 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:06.971629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:06.971633 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:06.971679 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:06.972241 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:06.972242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:06.972242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:06.972246 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:06.972259 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:06.972263 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:06.972299 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:06.972349 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:21:06.972360 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2875}
I20260928 23:21:06.983230 22191 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:06.983385 22191 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:06.990583 22191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb44b4c45e3a44e1a6fd49e7816ee8cd: Generated new TSK 0
I20260928 23:21:06.990780 22191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:21:07.001345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 22115
2026-09-28T23:21:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:21:07 dist-test-slave-djkr krb5kdc[22214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:21:07 dist-test-slave-djkr krb5kdc[22214](info): set up 2 sockets
Sep 28 23:21:07 dist-test-slave-djkr krb5kdc[22214](info): commencing operation
krb5kdc: starting...
W20260928 23:21:07.951352 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:08 dist-test-slave-djkr krb5kdc[22214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:21:08.060065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41913
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:08.137171 22227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:08.137360 22227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:08.137389 22227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:08.138489 22227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:08.138528 22227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:08.138546 22227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:08.138562 22227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:08.138576 22227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:08.140038 22227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:08.140265 22227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:08.140460 22227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:08.144869 22233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:08.144869 22235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:08.144960 22232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:08.145088 22227 server_base.cc:1034] running on GCE node
I20260928 23:21:08.145326 22227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:08.145551 22227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:08.146703 22227 hybrid_clock.cc:648] HybridClock initialized: now 1790637668146680 us; error 37 us; skew 500 ppm
Sep 28 23:21:08 dist-test-slave-djkr krb5kdc[22214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:08.153662 22227 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:21:08.154052 22227 webserver.cc:460] Webserver started at http://127.0.98.62:34623/ using document root <none> and password file <none>
I20260928 23:21:08.154258 22227 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:08.154300 22227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:08.154417 22227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:08.155396 22227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "81085226d8e14d65afa1a181865acb00"
format_stamp: "Formatted at 2026-09-28 23:21:08 on dist-test-slave-djkr"
server_key: "3434ed78a9cbf22f8d58b3fe4b35e479"
server_key_iv: "0c84ffe61c0e6867d450ab11282a7bf8"
server_key_version: "encryptionkey@0"
I20260928 23:21:08.155740 22227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "81085226d8e14d65afa1a181865acb00"
format_stamp: "Formatted at 2026-09-28 23:21:08 on dist-test-slave-djkr"
server_key: "3434ed78a9cbf22f8d58b3fe4b35e479"
server_key_iv: "0c84ffe61c0e6867d450ab11282a7bf8"
server_key_version: "encryptionkey@0"
I20260928 23:21:08.156925 22227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:21:08.157585 22242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:08.157711 22227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:21:08.157791 22227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "81085226d8e14d65afa1a181865acb00"
format_stamp: "Formatted at 2026-09-28 23:21:08 on dist-test-slave-djkr"
server_key: "3434ed78a9cbf22f8d58b3fe4b35e479"
server_key_iv: "0c84ffe61c0e6867d450ab11282a7bf8"
server_key_version: "encryptionkey@0"
I20260928 23:21:08.157850 22227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:08.185969 22227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:08.187830 22227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:08.187964 22227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:08.191514 22227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41913
I20260928 23:21:08.191587 22294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41913 every 8 connection(s)
I20260928 23:21:08.191901 22227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:08.192730 22295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:08.194144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 22227
I20260928 23:21:08.194253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:08.194435 392 external_mini_cluster.cc:1428] Setting key 1e1ec75283e1d805a77299d4611fce53
I20260928 23:21:08.195202 22295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00: Bootstrap starting.
I20260928 23:21:08.195895 22295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:08.196276 22295 log.cc:824] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:08.196967 22295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00: No bootstrap required, opened a new log
I20260928 23:21:08.198992 22295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81085226d8e14d65afa1a181865acb00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41913 } }
I20260928 23:21:08.199133 22295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:21:08.199168 22295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81085226d8e14d65afa1a181865acb00, State: Initialized, Role: FOLLOWER
I20260928 23:21:08.199354 22295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81085226d8e14d65afa1a181865acb00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41913 } }
I20260928 23:21:08.199452 22295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:21:08.199479 22295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:21:08.199527 22295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:08.200218 22295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81085226d8e14d65afa1a181865acb00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41913 } }
I20260928 23:21:08.200337 22295 leader_election.cc:302] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81085226d8e14d65afa1a181865acb00; no voters:
I20260928 23:21:08.200520 22295 leader_election.cc:288] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:21:08.200548 22300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:08.200722 22300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [term 1 LEADER]: Becoming Leader. State: Replica: 81085226d8e14d65afa1a181865acb00, State: Running, Role: LEADER
I20260928 23:21:08.200771 22295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:08.200810 22300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81085226d8e14d65afa1a181865acb00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41913 } }
I20260928 23:21:08.201136 22300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81085226d8e14d65afa1a181865acb00. Latest consensus state: current_term: 1 leader_uuid: "81085226d8e14d65afa1a181865acb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81085226d8e14d65afa1a181865acb00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41913 } } }
I20260928 23:21:08.201215 22300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:08.201185 22301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81085226d8e14d65afa1a181865acb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81085226d8e14d65afa1a181865acb00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41913 } } }
I20260928 23:21:08.201351 22301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00 [sys.catalog]: This master's current role is: LEADER
Sep 28 23:21:08 dist-test-slave-djkr krb5kdc[22214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260928 23:21:08.202481 22316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:08.203105 22316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:08.203176 22307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:08.203325 22307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:08.204675 22307 catalog_manager.cc:1347] Generated new cluster ID: d79fcb00647c4b1595145aff1d6537ae
I20260928 23:21:08.204727 22307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:21:08.205369 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:08.194927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43209 (local address 127.0.98.62:41913)
0928 23:21:08.195073 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:08.195080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:08.195105 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:21:08.195690 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:08.195700 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:08.195770 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:08.195921 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:08.196688 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:08.197620 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:08.198676 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:08.199073 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:08.199163 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:08.201979 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:08.201983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:08.201983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:08.201995 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:08.203723 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:08.204095 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:08.204098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:08.204099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:08.204201 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:08.204406 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:08.204413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:08.204414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:08.204708 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:08.204856 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:08.205072 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 23:21:08.205178 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260928 23:21:08.206171 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:08.210361 22307 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:08.210929 22307 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:08.214279 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:08.207500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45331 (local address 127.0.98.62:41913)
0928 23:21:08.207535 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:08.207536 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:08.207605 (+ 69us) server_negotiation.cc:407] Connection header received
0928 23:21:08.207660 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:08.207661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:08.207669 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:08.207733 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:08.208342 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:08.209050 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:08.210149 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:08.210486 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:08.210562 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:08.211801 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:08.211802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:08.211803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:08.211809 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:08.213250 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:08.213586 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:08.213587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:08.213587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:08.213645 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:08.213807 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:08.213807 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:08.213808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:08.213927 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:08.214023 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:08.214108 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 23:21:08.214193 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:21:08.219099 22307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81085226d8e14d65afa1a181865acb00: Generated new TSK 0
I20260928 23:21:08.219240 22307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:21:08.221364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 22227
2026-09-28T23:21:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:21:08 dist-test-slave-djkr krb5kdc[22326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:21:08 dist-test-slave-djkr krb5kdc[22326](info): set up 2 sockets
Sep 28 23:21:08 dist-test-slave-djkr krb5kdc[22326](info): commencing operation
krb5kdc: starting...
W20260928 23:21:09.167290 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:09 dist-test-slave-djkr krb5kdc[22326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:21:09.272352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:09.350103 22339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:09.350304 22339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:09.350334 22339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:09.351501 22339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:09.351562 22339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:09.351583 22339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:09.351600 22339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:09.351620 22339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:09.353104 22339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:09.353343 22339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:09.353538 22339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:09.358192 22344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:09.358254 22339 server_base.cc:1034] running on GCE node
W20260928 23:21:09.358206 22347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:09.358206 22345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:09.358609 22339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:09.358901 22339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:09.360039 22339 hybrid_clock.cc:648] HybridClock initialized: now 1790637669360020 us; error 31 us; skew 500 ppm
Sep 28 23:21:09 dist-test-slave-djkr krb5kdc[22326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:09.362372 22339 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:21:09.362778 22339 webserver.cc:460] Webserver started at http://127.0.98.62:41221/ using document root <none> and password file <none>
I20260928 23:21:09.362972 22339 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:09.363015 22339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:09.363128 22339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:09.364159 22339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "6b034fecf3d04a3595293f03a828979a"
format_stamp: "Formatted at 2026-09-28 23:21:09 on dist-test-slave-djkr"
server_key: "9f1e42c8e8c195e88d2efaa81f0edcf7"
server_key_iv: "c931486583ccc8286cc467776a8fc637"
server_key_version: "encryptionkey@0"
I20260928 23:21:09.364482 22339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b034fecf3d04a3595293f03a828979a"
format_stamp: "Formatted at 2026-09-28 23:21:09 on dist-test-slave-djkr"
server_key: "9f1e42c8e8c195e88d2efaa81f0edcf7"
server_key_iv: "c931486583ccc8286cc467776a8fc637"
server_key_version: "encryptionkey@0"
I20260928 23:21:09.365708 22339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:21:09.366376 22354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:09.366590 22339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:21:09.366675 22339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "6b034fecf3d04a3595293f03a828979a"
format_stamp: "Formatted at 2026-09-28 23:21:09 on dist-test-slave-djkr"
server_key: "9f1e42c8e8c195e88d2efaa81f0edcf7"
server_key_iv: "c931486583ccc8286cc467776a8fc637"
server_key_version: "encryptionkey@0"
I20260928 23:21:09.366736 22339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:09.378535 22339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:09.380484 22339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:09.380618 22339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:09.384090 22339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33235
I20260928 23:21:09.384133 22406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33235 every 8 connection(s)
I20260928 23:21:09.384433 22339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:09.385254 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:09.386052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 22339
I20260928 23:21:09.386132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:09.386302 392 external_mini_cluster.cc:1428] Setting key b53468e2c2ebbfc2a704d0823524f6dd
I20260928 23:21:09.388260 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a: Bootstrap starting.
I20260928 23:21:09.388942 22407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:09.389220 22407 log.cc:824] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:09.389832 22407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a: No bootstrap required, opened a new log
I20260928 23:21:09.391976 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:09.386747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55017 (local address 127.0.98.62:33235)
0928 23:21:09.386931 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:09.386940 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:09.386979 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:21:09.387427 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:09.387436 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:09.387499 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:09.387653 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:09.387839 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:09.389114 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:09.389886 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:09.390478 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:09.390578 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:09.391164 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:09.391168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:09.391169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:09.391378 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:09.391418 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:09.391434 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:09.391505 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:09.391647 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 23:21:09.391720 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260928 23:21:09.392076 22407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b034fecf3d04a3595293f03a828979a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260928 23:21:09.392215 22407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:21:09.392246 22407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b034fecf3d04a3595293f03a828979a, State: Initialized, Role: FOLLOWER
I20260928 23:21:09.392370 22407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b034fecf3d04a3595293f03a828979a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260928 23:21:09.392429 22407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:21:09.392473 22407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:21:09.392517 22407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:09.392910 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:09.393138 22407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b034fecf3d04a3595293f03a828979a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260928 23:21:09.393235 22407 leader_election.cc:302] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b034fecf3d04a3595293f03a828979a; no voters:
I20260928 23:21:09.393411 22407 leader_election.cc:288] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:21:09.393453 22412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:09.393617 22412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [term 1 LEADER]: Becoming Leader. State: Replica: 6b034fecf3d04a3595293f03a828979a, State: Running, Role: LEADER
I20260928 23:21:09.393643 22407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:09.394704 22412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b034fecf3d04a3595293f03a828979a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260928 23:21:09.395607 22414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b034fecf3d04a3595293f03a828979a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b034fecf3d04a3595293f03a828979a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } } }
I20260928 23:21:09.395676 22414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:09.395332 22413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b034fecf3d04a3595293f03a828979a. Latest consensus state: current_term: 1 leader_uuid: "6b034fecf3d04a3595293f03a828979a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b034fecf3d04a3595293f03a828979a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } } }
I20260928 23:21:09.395812 22413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a [sys.catalog]: This master's current role is: LEADER
W20260928 23:21:09.395919 22431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:09.395972 22431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:09.396027 22433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:09.396175 22433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:09.397619 22433 catalog_manager.cc:1347] Generated new cluster ID: 860da2e155fa48028ac4b94f40416fad
I20260928 23:21:09.397673 22433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:21:09.397780 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:09.394414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35595 (local address 127.0.98.62:33235)
0928 23:21:09.394449 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:09.394451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:09.394580 (+ 129us) server_negotiation.cc:407] Connection header received
0928 23:21:09.394638 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:09.394639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:09.394649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:09.394700 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:09.394856 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:09.395561 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:09.396494 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:09.396688 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:09.396748 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:09.397422 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:09.397423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:09.397423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:09.397427 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:09.397439 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:09.397443 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:09.397485 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:09.397700 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 23:21:09.397721 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:21:09.415174 22433 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:09.415742 22433 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:09.419410 22433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b034fecf3d04a3595293f03a828979a: Generated new TSK 0
I20260928 23:21:09.419595 22433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:21:09.421180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 22339
2026-09-28T23:21:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:21:09 dist-test-slave-djkr krb5kdc[22438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:21:09 dist-test-slave-djkr krb5kdc[22438](info): set up 2 sockets
Sep 28 23:21:09 dist-test-slave-djkr krb5kdc[22438](info): commencing operation
krb5kdc: starting...
W20260928 23:21:10.371462 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:10 dist-test-slave-djkr krb5kdc[22438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:21:10.480275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45371
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:10.558516 22451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:10.558707 22451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:10.558729 22451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:10.559862 22451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:10.559902 22451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:10.559914 22451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:10.559926 22451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:10.559937 22451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:10.561342 22451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:10.561569 22451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:10.561765 22451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:10.566483 22459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:10.566506 22456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:10.566519 22457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:10.566735 22451 server_base.cc:1034] running on GCE node
I20260928 23:21:10.566947 22451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:10.567180 22451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:10.568333 22451 hybrid_clock.cc:648] HybridClock initialized: now 1790637670568324 us; error 34 us; skew 500 ppm
Sep 28 23:21:10 dist-test-slave-djkr krb5kdc[22438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:10.570612 22451 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:21:10.570993 22451 webserver.cc:460] Webserver started at http://127.0.98.62:41753/ using document root <none> and password file <none>
I20260928 23:21:10.571189 22451 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:10.571233 22451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:10.571333 22451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:10.572427 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "39fbe1d900444367a63d34e06336d0ce"
format_stamp: "Formatted at 2026-09-28 23:21:10 on dist-test-slave-djkr"
server_key: "58f2f66dc4049bcee498757f1a3558b7"
server_key_iv: "dbee5c498384f2fd65a094ef92912e85"
server_key_version: "encryptionkey@0"
I20260928 23:21:10.572741 22451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "39fbe1d900444367a63d34e06336d0ce"
format_stamp: "Formatted at 2026-09-28 23:21:10 on dist-test-slave-djkr"
server_key: "58f2f66dc4049bcee498757f1a3558b7"
server_key_iv: "dbee5c498384f2fd65a094ef92912e85"
server_key_version: "encryptionkey@0"
I20260928 23:21:10.573940 22451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:21:10.574623 22466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:10.574774 22451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:21:10.574846 22451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "39fbe1d900444367a63d34e06336d0ce"
format_stamp: "Formatted at 2026-09-28 23:21:10 on dist-test-slave-djkr"
server_key: "58f2f66dc4049bcee498757f1a3558b7"
server_key_iv: "dbee5c498384f2fd65a094ef92912e85"
server_key_version: "encryptionkey@0"
I20260928 23:21:10.574910 22451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:10.577999 22451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:10.579656 22451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:10.579766 22451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:10.583303 22451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45371
I20260928 23:21:10.583348 22518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45371 every 8 connection(s)
I20260928 23:21:10.583668 22451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:10.584256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 22451
I20260928 23:21:10.584431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:10.584621 392 external_mini_cluster.cc:1428] Setting key 72d8dc47ee2eb1e4ceb25f55301f729d
I20260928 23:21:10.584722 22519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:10.587371 22519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce: Bootstrap starting.
I20260928 23:21:10.587764 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:10.585105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47993 (local address 127.0.98.62:45371)
0928 23:21:10.585951 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:10.585960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:10.585992 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:21:10.586504 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:10.586513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:10.586554 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:10.586698 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:10.587275 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:10.587278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:10.587279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:10.587283 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:10.587332 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:10.587346 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:10.587432 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:10.587522 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 23:21:10.587584 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":65,"threads_started":1}
I20260928 23:21:10.588043 22519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:10.588308 22519 log.cc:824] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:10.588651 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:10.588991 22519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce: No bootstrap required, opened a new log
I20260928 23:21:10.590878 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:10.589963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39175 (local address 127.0.98.62:45371)
0928 23:21:10.589998 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:10.590000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:10.590089 (+ 89us) server_negotiation.cc:407] Connection header received
0928 23:21:10.590156 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:10.590157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:10.590160 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:10.590205 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:10.590692 (+ 487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:10.590693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:10.590693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:21:10.590697 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:10.590709 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:21:10.590712 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:10.590757 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:10.590805 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:21:10.590817 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:21:10.590806 22519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fbe1d900444367a63d34e06336d0ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } }
I20260928 23:21:10.590953 22519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:21:10.590986 22519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fbe1d900444367a63d34e06336d0ce, State: Initialized, Role: FOLLOWER
I20260928 23:21:10.591109 22519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fbe1d900444367a63d34e06336d0ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } }
I20260928 23:21:10.591180 22519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:21:10.591210 22519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:21:10.591255 22519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:10.591825 22519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fbe1d900444367a63d34e06336d0ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } }
I20260928 23:21:10.591910 22519 leader_election.cc:302] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39fbe1d900444367a63d34e06336d0ce; no voters:
I20260928 23:21:10.592043 22519 leader_election.cc:288] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:21:10.592094 22529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:10.592260 22519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:10.592270 22529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [term 1 LEADER]: Becoming Leader. State: Replica: 39fbe1d900444367a63d34e06336d0ce, State: Running, Role: LEADER
I20260928 23:21:10.592442 22529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fbe1d900444367a63d34e06336d0ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } }
I20260928 23:21:10.592854 22531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39fbe1d900444367a63d34e06336d0ce. Latest consensus state: current_term: 1 leader_uuid: "39fbe1d900444367a63d34e06336d0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fbe1d900444367a63d34e06336d0ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } } }
I20260928 23:21:10.593060 22531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:10.592999 22530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39fbe1d900444367a63d34e06336d0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39fbe1d900444367a63d34e06336d0ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45371 } } }
I20260928 23:21:10.593235 22530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:10.593421 22538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:10.593793 22538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:10.595074 22538 catalog_manager.cc:1347] Generated new cluster ID: 5f5772c2ed1f4972b0b12efc6ca8113f
I20260928 23:21:10.595119 22538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:21:10.630201 22538 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:10.630383 22538 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:10.633487 22538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39fbe1d900444367a63d34e06336d0ce: Generated new TSK 0
I20260928 23:21:10.633669 22538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:21:10.646930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 22451
2026-09-28T23:21:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1225 ms)
[----------] 18 tests from AuthTokenIssuingTest (21712 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:21:10 dist-test-slave-djkr krb5kdc[22550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:21:10 dist-test-slave-djkr krb5kdc[22550](info): set up 2 sockets
Sep 28 23:21:10 dist-test-slave-djkr krb5kdc[22550](info): commencing operation
krb5kdc: starting...
W20260928 23:21:11.595269 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:11 dist-test-slave-djkr krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:21:11.706219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36709
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36709,127.0.98.61:33633,127.0.98.60:43513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:11.785115 22563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:21:11.785318 22563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:11.785341 22563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:11.785367 22563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:11.786468 22563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:11.786505 22563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:11.786523 22563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:11.786540 22563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:11.786556 22563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:11.788045 22563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36709,127.0.98.61:33633,127.0.98.60:43513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:11.788273 22563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:11.788468 22563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:11.792982 22571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:11.793061 22569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:11.792990 22568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:11.793159 22563 server_base.cc:1034] running on GCE node
I20260928 23:21:11.793406 22563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:11.793690 22563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:11.794837 22563 hybrid_clock.cc:648] HybridClock initialized: now 1790637671794818 us; error 34 us; skew 500 ppm
Sep 28 23:21:11 dist-test-slave-djkr krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:11.797154 22563 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:21:11.797562 22563 webserver.cc:460] Webserver started at http://127.0.98.62:36845/ using document root <none> and password file <none>
I20260928 23:21:11.797757 22563 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:11.797799 22563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:11.797912 22563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:11.798897 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "14afc0794661428ea06c36c4b8c8616e"
format_stamp: "Formatted at 2026-09-28 23:21:11 on dist-test-slave-djkr"
server_key: "76513c517b3d50dc92d47689ad401ee0"
server_key_iv: "478aceffd60f62dc3cda62a37ca4fff8"
server_key_version: "encryptionkey@0"
I20260928 23:21:11.799196 22563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "14afc0794661428ea06c36c4b8c8616e"
format_stamp: "Formatted at 2026-09-28 23:21:11 on dist-test-slave-djkr"
server_key: "76513c517b3d50dc92d47689ad401ee0"
server_key_iv: "478aceffd60f62dc3cda62a37ca4fff8"
server_key_version: "encryptionkey@0"
I20260928 23:21:11.800477 22563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:21:11.801122 22578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:11.801254 22563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:21:11.801326 22563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "14afc0794661428ea06c36c4b8c8616e"
format_stamp: "Formatted at 2026-09-28 23:21:11 on dist-test-slave-djkr"
server_key: "76513c517b3d50dc92d47689ad401ee0"
server_key_iv: "478aceffd60f62dc3cda62a37ca4fff8"
server_key_version: "encryptionkey@0"
I20260928 23:21:11.801389 22563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:11.808097 22563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:11.809975 22563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:11.810097 22563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:11.813690 22563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36709
I20260928 23:21:11.813800 22630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36709 every 8 connection(s)
I20260928 23:21:11.814038 22563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:11.814978 22631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:11.816565 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:11.820341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 22563
I20260928 23:21:11.820420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:11.820628 392 external_mini_cluster.cc:1428] Setting key 5c7b167b51177af6b8fe5ca3876a34ca
Sep 28 23:21:11 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:21:11.825613 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.817184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36709 (local address 127.0.0.1:58622)
0928 23:21:11.817331 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.817365 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:11.817838 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:11.818093 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:11.818095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:11.818346 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:11.818488 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:11.818492 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.819462 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.819462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:11.820152 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:11.820154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.820694 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.820696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:11.820758 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:11.821197 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:11.821205 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:11.822737 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:11.824300 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:11.824305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:11.824307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:11.824500 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:11.824695 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:11.824696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:11.824696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:11.824772 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:11.825082 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:11.825087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:11.825145 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 23:21:11.825223 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:21:11.825432 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":81,"thread_start_us":69,"threads_started":1}
I20260928 23:21:11.825616 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.817351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.0.98.62:36709)
0928 23:21:11.817463 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:11.817467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:11.817530 (+ 63us) server_negotiation.cc:407] Connection header received
0928 23:21:11.817983 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:11.817985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:11.818003 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:11.818077 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:11.818573 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.819373 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.820224 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.820584 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.820702 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:11.822807 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:11.822814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:11.822815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:11.822821 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:11.824227 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:11.824565 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:11.824568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:11.824568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:11.824643 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:11.824821 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:11.824822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:11.824822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:11.825024 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:11.825093 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:11.825251 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 23:21:11.825360 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20260928 23:21:11.826381 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:11.826689 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.826584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33633 (local address 127.0.0.1:53930)
0928 23:21:11.826624 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.826666 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 23:21:11.826807 22580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33633: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 23:21:11.827153 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.827071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33633 (local address 127.0.0.1:53942)
0928 23:21:11.827107 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.827130 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 23:21:11.827242 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33633: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33633: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 23:21:11.842722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33633
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36709,127.0.98.61:33633,127.0.98.60:43513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:21:11.876468 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } attempt: 1
I20260928 23:21:11.877019 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.876841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33633 (local address 127.0.0.1:53948)
0928 23:21:11.876932 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.876982 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 23:21:11.877413 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.877323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33633 (local address 127.0.0.1:53962)
0928 23:21:11.877365 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.877390 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:21:11.877506 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33633: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33633: connect: Connection refused (error 111)
W20260928 23:21:11.921337 22638 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:21:11.921552 22638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:11.921577 22638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:11.921602 22638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:11.922708 22638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:11.922746 22638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:11.922765 22638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:11.922781 22638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:11.922796 22638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:11.924280 22638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36709,127.0.98.61:33633,127.0.98.60:43513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:11.924530 22638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:11.924724 22638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:11.929212 22644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:11.929242 22646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:11.929256 22643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:11.929531 22638 server_base.cc:1034] running on GCE node
I20260928 23:21:11.929713 22638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:11.929982 22638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:11.931177 22638 hybrid_clock.cc:648] HybridClock initialized: now 1790637671931164 us; error 26 us; skew 500 ppm
Sep 28 23:21:11 dist-test-slave-djkr krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:11.933689 22638 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 23:21:11.934096 22638 webserver.cc:460] Webserver started at http://127.0.98.61:33921/ using document root <none> and password file <none>
I20260928 23:21:11.934305 22638 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:11.934358 22638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:11.934487 22638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:11.935495 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/data/instance:
uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0"
format_stamp: "Formatted at 2026-09-28 23:21:11 on dist-test-slave-djkr"
server_key: "758ef7185180db659cb8b410a1f9848c"
server_key_iv: "2e9fe567d55d0de23b3f577fe5b0d115"
server_key_version: "encryptionkey@0"
I20260928 23:21:11.935892 22638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/wal/instance:
uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0"
format_stamp: "Formatted at 2026-09-28 23:21:11 on dist-test-slave-djkr"
server_key: "758ef7185180db659cb8b410a1f9848c"
server_key_iv: "2e9fe567d55d0de23b3f577fe5b0d115"
server_key_version: "encryptionkey@0"
I20260928 23:21:11.937202 22638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:21:11.937906 22653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:11.938066 22638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:21:11.938149 22638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/wal
uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0"
format_stamp: "Formatted at 2026-09-28 23:21:11 on dist-test-slave-djkr"
server_key: "758ef7185180db659cb8b410a1f9848c"
server_key_iv: "2e9fe567d55d0de23b3f577fe5b0d115"
server_key_version: "encryptionkey@0"
I20260928 23:21:11.938210 22638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:11.945652 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } attempt: 2
I20260928 23:21:11.946074 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.945935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33633 (local address 127.0.0.1:53972)
0928 23:21:11.945992 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.946037 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 23:21:11.946425 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.946351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33633 (local address 127.0.0.1:53978)
0928 23:21:11.946386 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.946408 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 23:21:11.946563 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33633: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33633: connect: Connection refused (error 111)
I20260928 23:21:11.958240 22638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:11.960088 22638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:11.960210 22638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:11.963946 22638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33633
I20260928 23:21:11.963990 22705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33633 every 8 connection(s)
I20260928 23:21:11.964310 22638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
I20260928 23:21:11.965266 22706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:11.966749 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:11.967195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 22638
I20260928 23:21:11.967269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-1/wal/instance
I20260928 23:21:11.967440 392 external_mini_cluster.cc:1428] Setting key 5fa4dd327baaf14fb6929e3a8bd3aea6
Sep 28 23:21:11 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 23:21:11.975405 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.967419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.0.98.62:36709)
0928 23:21:11.967484 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:11.967486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:11.967721 (+ 235us) server_negotiation.cc:407] Connection header received
0928 23:21:11.968171 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:11.968173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:11.968189 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:11.968257 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:11.969068 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.969599 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.970568 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.970872 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.970955 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:11.972983 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:11.972985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:11.972985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:11.972993 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:11.974363 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:11.974699 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:11.974699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:11.974700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:11.974748 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:11.974936 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:11.974936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:11.974936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:11.975027 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:11.975072 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:11.975269 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 23:21:11.975341 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:21:11.975672 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.967424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36709 (local address 127.0.0.1:58630)
0928 23:21:11.967592 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.967632 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:11.968088 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:11.968406 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:11.968410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:11.968738 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:11.968951 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:11.968956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.969686 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.969687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:11.970473 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:11.970476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.970981 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.970983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:11.971091 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:11.971605 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:11.971614 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:11.972879 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:11.974433 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:11.974439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:11.974441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:11.974638 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:11.974796 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:11.974797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:11.974797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:11.974888 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:11.975069 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:11.975071 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:11.975179 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 23:21:11.975252 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:21:11.975418 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260928 23:21:11.976194 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 23:21:11 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:21:11.982955 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.976425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33633 (local address 127.0.0.1:53984)
0928 23:21:11.976467 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.976477 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:11.976512 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:11.976743 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:11.976744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:11.976854 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:11.976910 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:11.976912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.977964 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.977965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:11.978611 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:11.978613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.979052 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.979053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:11.979122 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:11.979440 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:11.979445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:11.980595 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:11.982063 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:11.982064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:11.982064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:11.982207 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:11.982371 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:11.982372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:11.982372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:11.982422 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:11.982652 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:11.982653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:11.982689 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 23:21:11.982748 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:21:11.982841 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:21:11.983021 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.976490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.0.98.61:33633)
0928 23:21:11.976623 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:11.976627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:11.976645 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:21:11.976665 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:11.976667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:11.976676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:11.976737 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:11.976979 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.977879 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.978687 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:11.978987 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:11.979038 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:11.980665 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:11.980672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:11.980672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:11.980678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:11.981991 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:11.982267 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:11.982271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:11.982271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:11.982320 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:11.982470 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:11.982471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:11.982471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:11.982595 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:11.982661 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:11.982785 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 23:21:11.982890 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260928 23:21:11.983510 22706 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:11.983832 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.983730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42264)
0928 23:21:11.983770 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.983811 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:21:11.983940 22654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
I20260928 23:21:11.984304 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:11.984222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42266)
0928 23:21:11.984258 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:11.984282 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 23:21:11.984464 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43513: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 23:21:11.991369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43513
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36709,127.0.98.61:33633,127.0.98.60:43513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:21:12.033718 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } attempt: 1
I20260928 23:21:12.034307 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.034156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42270)
0928 23:21:12.034225 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.034273 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 23:21:12.034686 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.034596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42272)
0928 23:21:12.034637 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.034666 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 23:21:12.034802 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43513: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
I20260928 23:21:12.037725 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } attempt: 3
Sep 28 23:21:12 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:21:12.044950 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.038058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33633 (local address 127.0.0.1:53986)
0928 23:21:12.038113 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.038125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:12.038181 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:12.038324 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:12.038324 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:12.038465 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:12.038551 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.038553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.039490 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.039491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.040052 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.040060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.040562 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.040563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.040621 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.041082 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:12.041086 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:12.042340 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:12.044060 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.044061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.044061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.044214 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.044406 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.044406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.044406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.044477 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.044713 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:12.044714 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:12.044752 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 23:21:12.044801 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:21:12.044871 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 23:21:12.045008 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.038100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.0.98.61:33633)
0928 23:21:12.038145 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.038148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.038165 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:21:12.038243 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.038244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.038256 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.038308 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.038610 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.039376 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.040130 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.040460 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.040527 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.042423 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.042425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.042425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.042433 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.043970 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.044277 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.044278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.044279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.044339 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.044527 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.044528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.044528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.044655 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.044717 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.044835 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.044919 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:21:12.045445 22631 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:12.045737 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.045653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42284)
0928 23:21:12.045691 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.045716 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 23:21:12.045992 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.045920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42290)
0928 23:21:12.045951 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.045972 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 23:21:12.046061 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43513: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
W20260928 23:21:12.072273 22713 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:21:12.072494 22713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:12.072518 22713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:12.072542 22713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:12.073688 22713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:12.073726 22713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:12.073745 22713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:12.073760 22713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:12.073774 22713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:12.075227 22713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36709,127.0.98.61:33633,127.0.98.60:43513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:12.075470 22713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:12.075696 22713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:21:12.077196 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } attempt: 1
I20260928 23:21:12.077656 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.077523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42306)
0928 23:21:12.077582 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.077625 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 23:21:12.077973 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.077894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42318)
0928 23:21:12.077931 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.077955 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 23:21:12.078075 22631 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43513: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
W20260928 23:21:12.080453 22718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:12.080489 22721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:12.080447 22719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:12.080585 22713 server_base.cc:1034] running on GCE node
I20260928 23:21:12.080861 22713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:12.081125 22713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:12.082270 22713 hybrid_clock.cc:648] HybridClock initialized: now 1790637672082255 us; error 28 us; skew 500 ppm
Sep 28 23:21:12 dist-test-slave-djkr krb5kdc[22550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:12.084640 22713 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 23:21:12.085011 22713 webserver.cc:460] Webserver started at http://127.0.98.60:42761/ using document root <none> and password file <none>
I20260928 23:21:12.085232 22713 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:12.085282 22713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:12.085397 22713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:12.086391 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/data/instance:
uuid: "4e9feef674b840f8b0b73d6eea6eee6a"
format_stamp: "Formatted at 2026-09-28 23:21:12 on dist-test-slave-djkr"
server_key: "fc78f32582e0ba6702df9a9b9ad88309"
server_key_iv: "43cc7dd324265817b613d37f11bc23db"
server_key_version: "encryptionkey@0"
I20260928 23:21:12.086712 22713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/wal/instance:
uuid: "4e9feef674b840f8b0b73d6eea6eee6a"
format_stamp: "Formatted at 2026-09-28 23:21:12 on dist-test-slave-djkr"
server_key: "fc78f32582e0ba6702df9a9b9ad88309"
server_key_iv: "43cc7dd324265817b613d37f11bc23db"
server_key_version: "encryptionkey@0"
I20260928 23:21:12.087916 22713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:21:12.088580 22728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:12.088752 22713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:21:12.088830 22713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/wal
uuid: "4e9feef674b840f8b0b73d6eea6eee6a"
format_stamp: "Formatted at 2026-09-28 23:21:12 on dist-test-slave-djkr"
server_key: "fc78f32582e0ba6702df9a9b9ad88309"
server_key_iv: "43cc7dd324265817b613d37f11bc23db"
server_key_version: "encryptionkey@0"
I20260928 23:21:12.088886 22713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:12.102944 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } attempt: 2
I20260928 23:21:12.103354 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.103236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42334)
0928 23:21:12.103294 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.103324 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 23:21:12.103720 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.103643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42340)
0928 23:21:12.103679 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.103704 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 23:21:12.103797 22706 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43513: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43513: connect: Connection refused (error 111)
I20260928 23:21:12.104687 22713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:12.106545 22713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:12.106663 22713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:12.110241 22713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43513
I20260928 23:21:12.110278 22780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43513 every 8 connection(s)
I20260928 23:21:12.110597 22713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
I20260928 23:21:12.111491 22781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:12.113001 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:12.115708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 22713
I20260928 23:21:12.115835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637417635540-392-0/minicluster-data/master-2/wal/instance
I20260928 23:21:12.116004 392 external_mini_cluster.cc:1428] Setting key d652d90fa8ca904d28f5b0b1b0f2a923
Sep 28 23:21:12 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 23:21:12 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:21:12.121703 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.113649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.0.98.62:36709)
0928 23:21:12.113709 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.113712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.113918 (+ 206us) server_negotiation.cc:407] Connection header received
0928 23:21:12.114294 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.114295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.114305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.114366 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.114943 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.115442 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.116665 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.116926 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.116995 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.118891 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.118893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.118893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.118901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.120436 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.120845 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.120846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.120846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.120915 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.121125 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.121125 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.121125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.121251 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.121314 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.121538 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.121611 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:21:12.121802 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.113613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36709 (local address 127.0.0.1:58636)
0928 23:21:12.113747 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.113781 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:12.114212 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:12.114421 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:12.114424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:12.114683 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:12.114869 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.114874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.115506 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.115507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.116585 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.116588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.117013 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.117014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.117111 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.117558 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:12.117565 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:12.118817 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:12.120514 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.120521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.120523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.120774 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.120978 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.120979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.120979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.121074 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.121304 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:12.121309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:12.121415 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 23:21:12.121494 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 23:21:12.121598 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260928 23:21:12.122442 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:12.124354 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.116354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.0.98.60:43513)
0928 23:21:12.116511 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.116515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.116627 (+ 112us) server_negotiation.cc:407] Connection header received
0928 23:21:12.116668 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.116670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.116689 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.116741 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.117372 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.118134 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.118926 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.119329 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.119397 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.121663 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.121668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.121669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.121675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.123324 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.123690 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.123693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.123693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.123753 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.123921 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.123922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.123922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.124042 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.124147 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.124233 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.124287 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
Sep 28 23:21:12 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 23:21:12 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:21:12.128954 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.122670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33633 (local address 127.0.0.1:53988)
0928 23:21:12.122713 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.122724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:12.122762 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:12.122900 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:12.122901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:12.123006 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:12.123031 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.123032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.123828 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.123829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.124506 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.124509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.124890 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.124890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.125028 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.125366 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:12.125371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:12.126571 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:12.128099 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.128100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.128100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.128282 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.128458 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.128458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.128459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.128503 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.128736 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:12.128736 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:12.128767 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 23:21:12.128807 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 23:21:12.128866 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:21:12.128990 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.122739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.0.98.61:33633)
0928 23:21:12.122791 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.122793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.122802 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:21:12.122825 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.122826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.122835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.122898 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.123089 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.123757 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.124570 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.124820 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.124887 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.126646 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.126647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.126647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.126653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.128025 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.128342 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.128343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.128343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.128398 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.128554 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.128555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.128555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.128680 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.128737 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.128831 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.128907 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:21:12.129288 22781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:12.131876 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.125139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.0.98.62:36709)
0928 23:21:12.125178 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.125180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.125188 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:21:12.125206 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.125206 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.125214 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.125253 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.125676 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.126109 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.126856 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.127074 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.127143 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.129336 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.129337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.129337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.129343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.130801 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.131152 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.131153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.131153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.131194 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.131352 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.131353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.131353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.131466 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.131680 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.131723 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.131789 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 23:21:12 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:21:12.136062 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.129491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42366)
0928 23:21:12.129528 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.129537 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:12.129569 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:12.129686 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:12.129687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:12.129813 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:12.129845 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.129846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.130641 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.130642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.131243 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.131245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.131689 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.131690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.131738 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.132085 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:12.132090 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:12.133321 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:12.135210 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.135211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.135211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.135353 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.135508 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.135509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.135509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.135577 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.135761 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:12.135762 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:12.135796 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 23:21:12.135855 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:21:12.135950 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:21:12.136108 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.129517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.0.98.60:43513)
0928 23:21:12.129549 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.129550 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.129561 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:21:12.129610 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.129611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.129619 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.129678 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.129906 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.130572 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.131306 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.131628 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.131684 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.133874 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.133875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.133875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.133880 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.135147 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.135406 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.135406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.135407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.135461 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.135619 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.135619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.135620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.135713 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.135763 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.135879 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.135994 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":15}
Sep 28 23:21:12 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:21:12.137491 22781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a: Bootstrap starting.
I20260928 23:21:12.138456 22781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:12.138751 22781 log.cc:824] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:12.139343 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.132728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.0.98.61:33633)
0928 23:21:12.132769 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.132771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.132809 (+ 38us) server_negotiation.cc:407] Connection header received
0928 23:21:12.132851 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.132852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.132860 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.132923 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.133379 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.133802 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.134539 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.134802 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.134870 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.136876 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.136878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.136878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.136884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.138363 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.138661 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.138662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.138662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.138719 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.138888 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.138889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.138889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.139000 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.139153 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.139192 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.139259 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:21:12.139485 22781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a: No bootstrap required, opened a new log
I20260928 23:21:12.140726 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:12.141256 22781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } }
I20260928 23:21:12.141376 22781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9feef674b840f8b0b73d6eea6eee6a, State: Initialized, Role: FOLLOWER
I20260928 23:21:12.141474 22781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } }
I20260928 23:21:12.141783 22781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:12.141728 22787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } } }
I20260928 23:21:12.141870 22787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:12.141567 22605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "14afc0794661428ea06c36c4b8c8616e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58650
W20260928 23:21:12.143327 22798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:12.143404 22798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:12.143687 22713 master_runner.cc:186] Error getting master registration for 127.0.98.62:36709: OK, instance_id { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" instance_seqno: 1790637671812657 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:21:12.146418 22713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:21:12.153201 22631 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } attempt: 2
Sep 28 23:21:12 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:21:12.160567 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.153543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42374 (local address 127.0.98.60:43513)
0928 23:21:12.153580 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.153582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.153590 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:21:12.153648 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.153649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.153658 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.153710 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.154015 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.154739 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.155715 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.156007 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.156059 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.157944 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.157946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.157946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.157953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.159426 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.159819 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.159820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.159821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.159880 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.160056 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.160057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.160057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.160214 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.160273 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.160407 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.160480 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:21:12.160571 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.153478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42374)
0928 23:21:12.153534 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.153547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:12.153596 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:12.153713 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:12.153714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:12.153894 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:12.153951 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.153953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.154807 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.154808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.155641 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.155644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.156064 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.156065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.156120 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.156487 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:12.156493 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:12.157866 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:12.159505 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.159506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.159507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.159753 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.159943 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.159943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.159944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.160004 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.160270 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:12.160270 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:12.160316 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 23:21:12.160380 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:21:12.160473 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 23:21:12.162146 22631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e: Bootstrap starting.
I20260928 23:21:12.162776 22631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:12.163003 22631 log.cc:824] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:12.163759 22631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e: No bootstrap required, opened a new log
I20260928 23:21:12.164791 22631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } }
I20260928 23:21:12.164907 22631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14afc0794661428ea06c36c4b8c8616e, State: Initialized, Role: FOLLOWER
I20260928 23:21:12.164973 22631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } }
I20260928 23:21:12.165197 22801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } } }
I20260928 23:21:12.165241 22631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:12.165282 22801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:12.165351 22605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 23:21:12.165421 22605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:12.166201 22605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } }
I20260928 23:21:12.166952 22605 leader_election.cc:288] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [CANDIDATE]: Term 1 election: Requested vote from peers 0d6c12c9ec3b4b8eb6318cdb134534b0 (127.0.98.61:33633), 4e9feef674b840f8b0b73d6eea6eee6a (127.0.98.60:43513)
W20260928 23:21:12.167109 22812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:12.167160 22812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:12.167375 22755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14afc0794661428ea06c36c4b8c8616e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4e9feef674b840f8b0b73d6eea6eee6a"
I20260928 23:21:12.167503 22755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:12.168404 22755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14afc0794661428ea06c36c4b8c8616e in term 1.
I20260928 23:21:12.168656 22579 leader_election.cc:302] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14afc0794661428ea06c36c4b8c8616e, 4e9feef674b840f8b0b73d6eea6eee6a; no voters:
I20260928 23:21:12.168761 22801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:12.168882 22801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [term 1 LEADER]: Becoming Leader. State: Replica: 14afc0794661428ea06c36c4b8c8616e, State: Running, Role: LEADER
I20260928 23:21:12.168956 22801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } }
I20260928 23:21:12.169344 22801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14afc0794661428ea06c36c4b8c8616e. Latest consensus state: current_term: 1 leader_uuid: "14afc0794661428ea06c36c4b8c8616e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } } }
I20260928 23:21:12.169457 22801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:12.168541 22563 master_runner.cc:186] Error getting master registration for 127.0.98.61:33633: OK, instance_id { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" instance_seqno: 1790637671962804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:21:12.170799 22563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:21:12.175717 22820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:12.176007 22820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:12.181535 22755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [term 1 FOLLOWER]: Refusing update from remote peer 14afc0794661428ea06c36c4b8c8616e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:21:12.181960 22801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:21:12.183298 22787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14afc0794661428ea06c36c4b8c8616e. Latest consensus state: current_term: 1 leader_uuid: "14afc0794661428ea06c36c4b8c8616e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } } }
I20260928 23:21:12.183354 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.171803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.0.98.62:36709)
0928 23:21:12.171860 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.171862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.172013 (+ 151us) server_negotiation.cc:407] Connection header received
0928 23:21:12.172174 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.172175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.172200 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.172231 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.173453 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.174170 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.175715 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.175970 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.176014 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.178478 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.178479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.178480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.178486 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.180212 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.180990 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.180992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.180992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.181061 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.182260 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.182275 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.182276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.182405 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.182501 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.183193 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.183270 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:21:12.183388 22787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:12.187932 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.172043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.0.98.60:43513)
0928 23:21:12.172077 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.172079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.176199 (+ 4120us) server_negotiation.cc:407] Connection header received
0928 23:21:12.177174 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.177175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.177186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.177214 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.177909 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.178630 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.179715 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.180014 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.180055 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.184527 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.184543 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.184543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.184551 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.186163 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.186791 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.186792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.186792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.186848 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.187143 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.187144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.187144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.187288 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.187371 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.187763 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.187842 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:21:12.187983 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.172223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.0.98.61:33633)
0928 23:21:12.172273 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.172275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.177030 (+ 4755us) server_negotiation.cc:407] Connection header received
0928 23:21:12.177103 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.177105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.177118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.177206 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.181760 (+ 4554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.182412 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.183337 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.183623 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.183665 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.185070 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.185071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.185072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.185078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.186780 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.187151 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.187152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.187153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.187206 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.187392 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.187392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.187393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.187516 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.187609 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.187799 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.187880 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:21:12.192816 22787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14afc0794661428ea06c36c4b8c8616e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } } }
I20260928 23:21:12.192902 22787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:12.193101 22801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14afc0794661428ea06c36c4b8c8616e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } } }
I20260928 23:21:12.193192 22801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:12.193306 22824 mvcc.cc:205] Tried to move back new op lower bound from 7334451905253208064 to 7334451905204654080. Current Snapshot: MvccSnapshot[applied={T|T < 7334451905253208064}]
I20260928 23:21:12.194855 22706 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } attempt: 3
I20260928 23:21:12.195793 22825 mvcc.cc:205] Tried to move back new op lower bound from 7334451905253208064 to 7334451905204654080. Current Snapshot: MvccSnapshot[applied={T|T < 7334451905253208064}]
Sep 28 23:21:12 dist-test-slave-djkr krb5kdc[22550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:21:12.199844 22820 catalog_manager.cc:1347] Generated new cluster ID: 01ee90c283754188bb1dc67241d113e6
I20260928 23:21:12.199891 22820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:21:12.202802 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.195116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43513 (local address 127.0.0.1:42392)
0928 23:21:12.195212 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:12.195222 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:12.195260 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:12.195398 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:12.195399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:12.195586 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:12.195635 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.195638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.196529 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.196530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.197263 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:12.197266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.197733 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.197733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:12.197793 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.198245 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:12.198251 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:12.199703 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:12.201497 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.201498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.201499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.201713 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.201998 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:12.201998 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:12.201999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:12.202069 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:12.202496 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:12.202497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:12.202558 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 23:21:12.202603 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:21:12.202698 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 23:21:12.204234 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:12.195171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.0.98.60:43513)
0928 23:21:12.195233 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.195235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.195248 (+ 13us) server_negotiation.cc:407] Connection header received
0928 23:21:12.195308 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.195309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.195332 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.195389 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.195714 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.196447 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.197346 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:12.197664 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:12.197729 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:12.199779 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:12.199781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:12.199781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:12.199788 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:12.201379 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.201796 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.201798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.201798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.201886 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:12.202138 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:12.202139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:12.202139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:12.202348 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:12.203279 (+ 931us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:12.203389 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 23:21:12.204082 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:21:12.206004 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0: Bootstrap starting.
I20260928 23:21:12.206715 22706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:12.206995 22706 log.cc:824] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:12.211694 22820 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:12.212414 22820 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:12.213274 22706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0: No bootstrap required, opened a new log
I20260928 23:21:12.215258 22706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } }
I20260928 23:21:12.215394 22706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6c12c9ec3b4b8eb6318cdb134534b0, State: Initialized, Role: FOLLOWER
I20260928 23:21:12.215479 22706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } }
I20260928 23:21:12.216269 22679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:12.216292 22680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14afc0794661428ea06c36c4b8c8616e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0"
I20260928 23:21:12.217567 22679 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [term 1 FOLLOWER]: Refusing update from remote peer 14afc0794661428ea06c36c4b8c8616e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:21:12.218592 22680 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14afc0794661428ea06c36c4b8c8616e in term 1.
I20260928 23:21:12.218878 22818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:21:12.216190 22706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:12.221580 22820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e: Generated new TSK 0
I20260928 23:21:12.221748 22820 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:21:12.222862 22846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:12.222911 22846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:12.227629 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14afc0794661428ea06c36c4b8c8616e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } } }
I20260928 23:21:12.227712 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:12.227761 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14afc0794661428ea06c36c4b8c8616e. Latest consensus state: current_term: 1 leader_uuid: "14afc0794661428ea06c36c4b8c8616e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } } }
I20260928 23:21:12.227799 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:12.227826 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14afc0794661428ea06c36c4b8c8616e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } } }
I20260928 23:21:12.227859 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:12.228394 22833 mvcc.cc:205] Tried to move back new op lower bound from 7334451905363312640 to 7334451905204654080. Current Snapshot: MvccSnapshot[applied={T|T < 7334451905253208064}]
I20260928 23:21:12.229176 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14afc0794661428ea06c36c4b8c8616e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14afc0794661428ea06c36c4b8c8616e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36709 } } peers { permanent_uuid: "0d6c12c9ec3b4b8eb6318cdb134534b0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33633 } } peers { permanent_uuid: "4e9feef674b840f8b0b73d6eea6eee6a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43513 } } }
I20260928 23:21:12.229265 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14afc0794661428ea06c36c4b8c8616e [sys.catalog]: This master's current role is: LEADER
W20260928 23:21:12.251843 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.250196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.0.98.61:33633)
0928 23:21:12.250242 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.250244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.251161 (+ 917us) server_negotiation.cc:407] Connection header received
0928 23:21:12.251264 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.251265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.251275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.251304 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.251759 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54010: BlockingRecv error: recv got EOF from 127.0.0.1:54010 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 23:21:12.252600 22634 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.250612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.0.98.62:36709)
0928 23:21:12.250653 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.250655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.252099 (+ 1444us) server_negotiation.cc:407] Connection header received
0928 23:21:12.252196 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.252198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.252208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.252236 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.252540 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58664: BlockingRecv error: recv got EOF from 127.0.0.1:58664 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 23:21:12.253304 22785 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.250897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.0.98.60:43513)
0928 23:21:12.250948 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.250950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.252817 (+ 1867us) server_negotiation.cc:407] Connection header received
0928 23:21:12.252908 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.252909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.252918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.252957 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.253247 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42398: BlockingRecv error: recv got EOF from 127.0.0.1:42398 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 23:21:12.260568 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.259783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.0.98.61:33633)
0928 23:21:12.259820 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.259822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.260056 (+ 234us) server_negotiation.cc:407] Connection header received
0928 23:21:12.260159 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.260160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.260178 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.260205 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.260516 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54012: BlockingRecv error: recv got EOF from 127.0.0.1:54012 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 23:21:12.263593 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.262800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.0.98.61:33633)
0928 23:21:12.262834 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.262836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.263063 (+ 227us) server_negotiation.cc:407] Connection header received
0928 23:21:12.263149 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.263150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.263159 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.263185 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.263520 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54022: BlockingRecv error: recv got EOF from 127.0.0.1:54022 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 23:21:12.270519 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.269729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.0.98.61:33633)
0928 23:21:12.269793 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.269795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.270036 (+ 241us) server_negotiation.cc:407] Connection header received
0928 23:21:12.270125 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.270126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.270136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.270163 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.270473 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54030: BlockingRecv error: recv got EOF from 127.0.0.1:54030 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 23:21:12.276510 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.275733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.0.98.61:33633)
0928 23:21:12.275767 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.275769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.276013 (+ 244us) server_negotiation.cc:407] Connection header received
0928 23:21:12.276109 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.276109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.276120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.276146 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.276462 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54046: BlockingRecv error: recv got EOF from 127.0.0.1:54046 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 23:21:12.286463 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.285694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.0.98.61:33633)
0928 23:21:12.285731 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.285733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.285964 (+ 231us) server_negotiation.cc:407] Connection header received
0928 23:21:12.286054 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.286055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.286065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.286092 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.286410 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54054: BlockingRecv error: recv got EOF from 127.0.0.1:54054 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 23:21:12.304320 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.303602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.0.98.61:33633)
0928 23:21:12.303643 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.303645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.303802 (+ 157us) server_negotiation.cc:407] Connection header received
0928 23:21:12.303897 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.303898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.303910 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.303953 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.304262 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54058: BlockingRecv error: recv got EOF from 127.0.0.1:54058 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 23:21:12.337921 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.337269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.0.98.61:33633)
0928 23:21:12.337315 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.337317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.337448 (+ 131us) server_negotiation.cc:407] Connection header received
0928 23:21:12.337503 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.337504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.337516 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.337561 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.337858 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54064: BlockingRecv error: recv got EOF from 127.0.0.1:54064 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 23:21:12.403697 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.402992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.0.98.61:33633)
0928 23:21:12.403081 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.403083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.403109 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:21:12.403230 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.403232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.403243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.403300 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.403640 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54070: BlockingRecv error: recv got EOF from 127.0.0.1:54070 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 23:21:12.533495 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.532734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.0.98.61:33633)
0928 23:21:12.532801 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.532803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.532914 (+ 111us) server_negotiation.cc:407] Connection header received
0928 23:21:12.532993 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.532994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.533006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.533050 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.533432 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54080: BlockingRecv error: recv got EOF from 127.0.0.1:54080 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 23:21:12.791441 22710 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:12.790756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.0.98.61:33633)
0928 23:21:12.790811 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:12.790812 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:12.790868 (+ 56us) server_negotiation.cc:407] Connection header received
0928 23:21:12.790959 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:12.790960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:12.790971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:12.791013 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:12.791381 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54090: BlockingRecv error: recv got EOF from 127.0.0.1:54090 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:21:13.143831 22798 catalog_manager.cc:1259] Loaded cluster ID: 01ee90c283754188bb1dc67241d113e6
I20260928 23:21:13.143891 22798 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a: loading cluster ID for follower catalog manager: success
I20260928 23:21:13.144695 22798 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a: acquiring CA information for follower catalog manager: success
I20260928 23:21:13.144975 22798 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e9feef674b840f8b0b73d6eea6eee6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:21:13.223315 22846 catalog_manager.cc:1259] Loaded cluster ID: 01ee90c283754188bb1dc67241d113e6
I20260928 23:21:13.223387 22846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0: loading cluster ID for follower catalog manager: success
I20260928 23:21:13.224259 22846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0: acquiring CA information for follower catalog manager: success
I20260928 23:21:13.224582 22846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d6c12c9ec3b4b8eb6318cdb134534b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:21:13.306999 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:13.304538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.0.98.61:33633)
0928 23:21:13.304701 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:13.304704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:13.304716 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:21:13.304745 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:13.304746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:13.304797 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:13.304868 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:21:13.305047 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:13.305545 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:13.306317 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:13.306523 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:13.306580 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:13.306705 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:21:13.306815 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:21:13.306886 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:13.306898 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 23:21:13.306916 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260928 23:21:13.310987 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:13.308738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.0.98.60:43513)
0928 23:21:13.308830 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:13.308831 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:13.308839 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:21:13.308878 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:13.308879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:13.308913 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:13.308986 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:21:13.309061 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:13.309563 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:13.310291 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:13.310510 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:13.310560 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:13.310713 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:21:13.310803 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:21:13.310855 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:13.310876 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 23:21:13.310907 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":47,"threads_started":1}
I20260928 23:21:13.315063 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:13.312692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.0.98.60:43513)
0928 23:21:13.312729 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:13.312731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:13.312738 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:21:13.312790 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:13.312791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:13.312800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:13.312848 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:21:13.312982 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:13.313435 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:13.314377 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:13.314633 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:13.314689 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:13.314808 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:21:13.314834 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:21:13.314873 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:13.314983 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 23:21:13.315002 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:21:13.315227 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:13.312468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.0.98.61:33633)
0928 23:21:13.312510 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:13.312512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:13.312676 (+ 164us) server_negotiation.cc:407] Connection header received
0928 23:21:13.312701 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:13.312702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:13.312711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:13.312755 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:21:13.312915 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:13.313641 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:13.314561 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:13.314797 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:13.314854 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:13.314988 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:21:13.315020 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:21:13.315048 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:13.315144 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:21:13.315163 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:21:13.315390 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:13.312505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.0.98.62:36709)
0928 23:21:13.312591 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:13.312593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:13.312740 (+ 147us) server_negotiation.cc:407] Connection header received
0928 23:21:13.312783 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:13.312784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:13.312831 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:13.312898 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:21:13.313044 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:13.313772 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:13.314583 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:13.314862 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:13.314919 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:13.315090 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:21:13.315189 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:21:13.315244 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:13.315293 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 23:21:13.315317 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":39,"threads_started":1}
I20260928 23:21:13.321079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 22563
I20260928 23:21:13.325961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 22638
I20260928 23:21:13.330096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 22713
2026-09-28T23:21:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2683 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:21:13 dist-test-slave-djkr krb5kdc[22932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:21:13 dist-test-slave-djkr krb5kdc[22932](info): set up 2 sockets
Sep 28 23:21:13 dist-test-slave-djkr krb5kdc[22932](info): commencing operation
krb5kdc: starting...
W20260928 23:21:14.275506 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/test-user.keytab.
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:21:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:21:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:21:14.384516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40679
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40679,127.0.98.61:33731,127.0.98.60:38235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:21:14.462357 22945 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:21:14.462559 22945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:14.462574 22945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:14.462592 22945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:14.463676 22945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:14.463708 22945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:14.463721 22945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:14.463732 22945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:14.463742 22945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:14.465144 22945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40679,127.0.98.61:33731,127.0.98.60:38235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:14.465320 22945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:14.465483 22945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:21:14.470275 22945 server_base.cc:1034] running on GCE node
W20260928 23:21:14.470247 22951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:14.470261 22953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:14.470243 22950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:14.470834 22945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:14.471127 22945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:14.472268 22945 hybrid_clock.cc:648] HybridClock initialized: now 1790637674472255 us; error 30 us; skew 500 ppm
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:14.474635 22945 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:21:14.475018 22945 webserver.cc:460] Webserver started at http://127.0.98.62:36141/ using document root <none> and password file <none>
I20260928 23:21:14.475207 22945 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:14.475255 22945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:14.475376 22945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:14.476449 22945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/data/instance:
uuid: "0713b764e0844703beb3d045b1b3bd45"
format_stamp: "Formatted at 2026-09-28 23:21:14 on dist-test-slave-djkr"
server_key: "3974c171b70225a9571b30726f68edc1"
server_key_iv: "8ebabc28186f318004ef2b75bf8ce45a"
server_key_version: "encryptionkey@0"
I20260928 23:21:14.476784 22945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/wal/instance:
uuid: "0713b764e0844703beb3d045b1b3bd45"
format_stamp: "Formatted at 2026-09-28 23:21:14 on dist-test-slave-djkr"
server_key: "3974c171b70225a9571b30726f68edc1"
server_key_iv: "8ebabc28186f318004ef2b75bf8ce45a"
server_key_version: "encryptionkey@0"
I20260928 23:21:14.478061 22945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:21:14.478808 22960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:14.479010 22945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:21:14.479099 22945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/wal
uuid: "0713b764e0844703beb3d045b1b3bd45"
format_stamp: "Formatted at 2026-09-28 23:21:14 on dist-test-slave-djkr"
server_key: "3974c171b70225a9571b30726f68edc1"
server_key_iv: "8ebabc28186f318004ef2b75bf8ce45a"
server_key_version: "encryptionkey@0"
I20260928 23:21:14.479166 22945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:14.506307 22945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:14.508545 22945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:14.508688 22945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:14.512364 22945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40679
I20260928 23:21:14.512447 23012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40679 every 8 connection(s)
I20260928 23:21:14.512741 22945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:21:14.513717 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:14.515280 23013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:14.518999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 22945
I20260928 23:21:14.519102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-0/wal/instance
I20260928 23:21:14.519284 392 external_mini_cluster.cc:1428] Setting key 135eeb5b9d280f837d311a584542c7eb
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:21:14.524905 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.515939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40679 (local address 127.0.0.1:41468)
0928 23:21:14.516087 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.516124 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:14.516576 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:14.516820 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:14.516822 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:14.517066 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:14.517222 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.517226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.518171 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.518172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.518910 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.518912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.519480 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.519481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.519580 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.520071 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:14.520082 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:14.521778 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:14.523581 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.523587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.523590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.523768 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.523968 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.523968 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.523969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.524039 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.524362 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:14.524367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:14.524430 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 23:21:14.524507 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:21:14.524724 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":77,"thread_start_us":68,"threads_started":1}
I20260928 23:21:14.524909 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.515961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.0.98.62:40679)
0928 23:21:14.516124 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.516134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.516207 (+ 73us) server_negotiation.cc:407] Connection header received
0928 23:21:14.516708 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.516710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.516726 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.516808 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.517286 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.518097 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.518996 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.519362 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.519451 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.521856 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.521863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.521863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.521869 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.523494 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.523836 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.523839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.523839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.523916 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.524092 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.524093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.524093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.524295 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.524375 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.524535 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.524652 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260928 23:21:14.525589 23013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:14.525908 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.525808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33731 (local address 127.0.0.1:46282)
0928 23:21:14.525845 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.525885 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 23:21:14.526055 22963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33731: connect: Connection refused (error 111)
I20260928 23:21:14.526420 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.526358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33731 (local address 127.0.0.1:46286)
0928 23:21:14.526383 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.526404 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
W20260928 23:21:14.526507 23013 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33731: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33731: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 23:21:14.543043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33731
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40679,127.0.98.61:33731,127.0.98.60:38235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:21:14.575711 23013 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } attempt: 1
I20260928 23:21:14.576241 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.576097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33731 (local address 127.0.0.1:46296)
0928 23:21:14.576157 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.576206 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 23:21:14.576571 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.576480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33731 (local address 127.0.0.1:46298)
0928 23:21:14.576521 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.576549 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:21:14.576664 23013 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33731: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33731: connect: Connection refused (error 111)
W20260928 23:21:14.621523 23020 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:21:14.621762 23020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:14.621783 23020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:14.621814 23020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:14.622916 23020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:14.622952 23020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:14.622964 23020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:14.622975 23020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:14.622987 23020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:14.624472 23020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40679,127.0.98.61:33731,127.0.98.60:38235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:14.624698 23020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:14.624894 23020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:14.629369 23025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:14.629465 23020 server_base.cc:1034] running on GCE node
W20260928 23:21:14.629547 23028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:14.629426 23026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:14.629799 23020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:14.630059 23020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:14.631191 23020 hybrid_clock.cc:648] HybridClock initialized: now 1790637674631180 us; error 28 us; skew 500 ppm
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:14.633554 23020 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 23:21:14.633910 23020 webserver.cc:460] Webserver started at http://127.0.98.61:46077/ using document root <none> and password file <none>
I20260928 23:21:14.634096 23020 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:14.634135 23020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:14.634224 23020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:14.635183 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/data/instance:
uuid: "3a9afaa74caa42abaf47468e1fe624a2"
format_stamp: "Formatted at 2026-09-28 23:21:14 on dist-test-slave-djkr"
server_key: "2176ed5b1d6ef1868cf4da043556430c"
server_key_iv: "c929f1e730772379f619a39360689f3e"
server_key_version: "encryptionkey@0"
I20260928 23:21:14.635464 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/wal/instance:
uuid: "3a9afaa74caa42abaf47468e1fe624a2"
format_stamp: "Formatted at 2026-09-28 23:21:14 on dist-test-slave-djkr"
server_key: "2176ed5b1d6ef1868cf4da043556430c"
server_key_iv: "c929f1e730772379f619a39360689f3e"
server_key_version: "encryptionkey@0"
I20260928 23:21:14.636667 23020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:21:14.637271 23035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:14.637408 23020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:21:14.637470 23020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/wal
uuid: "3a9afaa74caa42abaf47468e1fe624a2"
format_stamp: "Formatted at 2026-09-28 23:21:14 on dist-test-slave-djkr"
server_key: "2176ed5b1d6ef1868cf4da043556430c"
server_key_iv: "c929f1e730772379f619a39360689f3e"
server_key_version: "encryptionkey@0"
I20260928 23:21:14.637535 23020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:14.644805 23013 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } attempt: 2
I20260928 23:21:14.645176 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.645070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33731 (local address 127.0.0.1:46308)
0928 23:21:14.645121 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.645148 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:21:14.645464 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.645388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33731 (local address 127.0.0.1:46322)
0928 23:21:14.645423 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.645446 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 23:21:14.645537 23013 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33731: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33731: connect: Connection refused (error 111)
I20260928 23:21:14.648437 23020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:14.650030 23020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:14.650149 23020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:14.653715 23020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33731
I20260928 23:21:14.653752 23087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33731 every 8 connection(s)
I20260928 23:21:14.654049 23020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/data/info.pb
I20260928 23:21:14.654950 23088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:14.656440 23088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:14.657295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 23020
I20260928 23:21:14.657402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-1/wal/instance
I20260928 23:21:14.657591 392 external_mini_cluster.cc:1428] Setting key 0b5cc7713744dbaca6def02e1f7c6926
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 23:21:14.665592 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.657027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.0.98.62:40679)
0928 23:21:14.657094 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.657097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.657317 (+ 220us) server_negotiation.cc:407] Connection header received
0928 23:21:14.657954 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.657955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.657967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.658015 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.658779 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.659267 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.660532 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.660735 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.660794 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.663069 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.663070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.663070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.663077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.664377 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.664799 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.664800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.664800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.664843 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.665100 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.665101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.665101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.665203 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.665248 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.665470 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.665530 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 23:21:14.665762 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.657001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40679 (local address 127.0.0.1:41474)
0928 23:21:14.657185 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.657230 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:14.657853 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:14.658098 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:14.658102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:14.658414 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:14.658673 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.658679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.659376 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.659377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.660447 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.660450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.660826 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.660827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.660985 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.661439 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:14.661449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:14.662981 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:14.664457 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.664464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.664467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.664733 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.664898 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.664899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.664900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.665040 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.665256 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:14.665261 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:14.665373 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 23:21:14.665455 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:21:14.665583 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260928 23:21:14.666292 23088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:21:14.672983 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.666560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33731 (local address 127.0.0.1:46336)
0928 23:21:14.666606 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.666615 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:14.666648 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:14.666879 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:14.666880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:14.667019 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:14.667065 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.667067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.667997 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.667998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.668499 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.668501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.669057 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.669058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.669119 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.669456 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:14.669460 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:14.670521 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:14.672086 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.672087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.672087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.672229 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.672394 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.672395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.672395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.672445 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.672677 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:14.672678 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:14.672714 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 23:21:14.672779 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 23:21:14.672875 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 23:21:14.673043 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.666614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.0.98.61:33731)
0928 23:21:14.666746 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.666751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.666768 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:21:14.666787 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.666789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.666799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.666857 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.667122 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.667888 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.668593 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.668989 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.669050 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.670592 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.670600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.670600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.670606 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.672015 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.672292 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.672295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.672295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.672343 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.672493 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.672493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.672494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.672619 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.672686 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.672813 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.672926 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260928 23:21:14.673547 23088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:14.673882 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.673778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52298)
0928 23:21:14.673817 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.673858 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 23:21:14.674003 23036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
I20260928 23:21:14.674345 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.674271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52314)
0928 23:21:14.674309 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.674330 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:21:14.674428 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38235: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 23:21:14.680001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNOOpNT/build/release/bin/kudu
/tmp/dist-test-taskNOOpNT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38235
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40679,127.0.98.61:33731,127.0.98.60:38235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:21:14.723644 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } attempt: 1
I20260928 23:21:14.724157 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.724002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52326)
0928 23:21:14.724076 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.724122 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 23:21:14.724558 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.724470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52336)
0928 23:21:14.724511 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.724538 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 23:21:14.724661 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38235: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
I20260928 23:21:14.736680 23013 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } attempt: 3
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:21:14.743778 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.737053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33731 (local address 127.0.0.1:46348)
0928 23:21:14.737101 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.737114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:14.737175 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:14.737339 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:14.737340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:14.737491 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:14.737556 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.737558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.738481 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.738481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.739010 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.739011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.739480 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.739481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.739540 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.739989 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:14.739993 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:14.741159 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:14.742920 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.742928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.742928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.743082 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.743261 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.743261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.743262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.743305 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.743519 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:14.743519 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:14.743588 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 23:21:14.743640 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:21:14.743710 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 23:21:14.743830 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.737098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.0.98.61:33731)
0928 23:21:14.737163 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.737165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.737177 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:21:14.737225 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.737226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.737240 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.737301 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.737627 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.738395 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.739084 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.739409 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.739479 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.741233 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.741234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.741235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.741243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.742838 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.743146 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.743147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.743148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.743208 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.743346 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.743347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.743347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.743465 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.743525 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.743668 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.743744 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:21:14.744293 23013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:14.744623 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.744537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52340)
0928 23:21:14.744578 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.744603 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:21:14.744913 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.744834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52342)
0928 23:21:14.744870 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.744893 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:21:14.745002 23013 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38235: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
W20260928 23:21:14.759130 23095 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:21:14.759316 23095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:21:14.759341 23095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:21:14.759366 23095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:21:14.760529 23095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:21:14.760568 23095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:21:14.760587 23095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:21:14.760602 23095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:21:14.760618 23095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:21:14.762050 23095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40679,127.0.98.61:33731,127.0.98.60:38235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:21:14.762276 23095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:21:14.762475 23095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:21:14.767038 23101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:14.767011 23103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:21:14.767089 23100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:21:14.767201 23095 server_base.cc:1034] running on GCE node
I20260928 23:21:14.767422 23095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:21:14.767717 23095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:21:14.768854 23095 hybrid_clock.cc:648] HybridClock initialized: now 1790637674768846 us; error 24 us; skew 500 ppm
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:21:14.771039 23095 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 23:21:14.771436 23095 webserver.cc:460] Webserver started at http://127.0.98.60:43145/ using document root <none> and password file <none>
I20260928 23:21:14.771688 23095 fs_manager.cc:359] Metadata directory not provided
I20260928 23:21:14.771734 23095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:21:14.771848 23095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:21:14.772832 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/data/instance:
uuid: "880687de171345f2a46d82abacca21fa"
format_stamp: "Formatted at 2026-09-28 23:21:14 on dist-test-slave-djkr"
server_key: "da9a09f50fc5863b7486cf5207fad191"
server_key_iv: "35176a38cc2409945f39b208712d6f9d"
server_key_version: "encryptionkey@0"
I20260928 23:21:14.773137 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/wal/instance:
uuid: "880687de171345f2a46d82abacca21fa"
format_stamp: "Formatted at 2026-09-28 23:21:14 on dist-test-slave-djkr"
server_key: "da9a09f50fc5863b7486cf5207fad191"
server_key_iv: "35176a38cc2409945f39b208712d6f9d"
server_key_version: "encryptionkey@0"
I20260928 23:21:14.774305 23095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:21:14.774917 23110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:21:14.775053 23095 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:21:14.775130 23095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/wal
uuid: "880687de171345f2a46d82abacca21fa"
format_stamp: "Formatted at 2026-09-28 23:21:14 on dist-test-slave-djkr"
server_key: "da9a09f50fc5863b7486cf5207fad191"
server_key_iv: "35176a38cc2409945f39b208712d6f9d"
server_key_version: "encryptionkey@0"
I20260928 23:21:14.775187 23095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:21:14.776124 23013 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } attempt: 1
I20260928 23:21:14.776458 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.776328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52346)
0928 23:21:14.776373 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.776402 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 23:21:14.776732 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.776655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52350)
0928 23:21:14.776688 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.776710 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 23:21:14.776808 23013 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38235: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
I20260928 23:21:14.792811 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } attempt: 2
I20260928 23:21:14.793215 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.793076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52366)
0928 23:21:14.793136 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.793178 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 23:21:14.793562 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.793479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52382)
0928 23:21:14.793518 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.793542 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:21:14.793648 23088 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38235: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38235: connect: Connection refused (error 111)
I20260928 23:21:14.797678 23095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:21:14.799377 23095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:21:14.799477 23095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:21:14.803263 23095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38235
I20260928 23:21:14.803293 23162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38235 every 8 connection(s)
I20260928 23:21:14.803669 23095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/data/info.pb
I20260928 23:21:14.804088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNOOpNT/build/release/bin/kudu as pid 23095
I20260928 23:21:14.804179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNOOpNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637417635540-392-0/minicluster-data/master-2/wal/instance
I20260928 23:21:14.804373 392 external_mini_cluster.cc:1428] Setting key f0b023df25efac115eace5782dd0fbbb
I20260928 23:21:14.804713 23163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:21:14.806862 23163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:21:14.814153 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.807574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.0.98.62:40679)
0928 23:21:14.807621 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.807623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.807705 (+ 82us) server_negotiation.cc:407] Connection header received
0928 23:21:14.807762 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.807763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.807771 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.807815 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.808290 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.808743 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.809484 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.809750 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.809820 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.811954 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.811955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.811955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.811963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.813083 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.813413 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.813414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.813414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.813463 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.813658 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.813658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.813659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.813760 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.813808 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.813990 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.814067 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:21:14.814342 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.804827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.0.98.60:38235)
0928 23:21:14.805006 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.805015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.805050 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:21:14.805796 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.805806 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.805874 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.806028 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.806601 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.807948 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.808665 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.809091 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.809192 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.811518 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.811526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.811526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.811534 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.813076 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.813400 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.813403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.813403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.813483 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.813646 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.813647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.813647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.813880 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.814008 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.814089 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.814167 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260928 23:21:14.814344 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.807458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40679 (local address 127.0.0.1:41478)
0928 23:21:14.807605 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.807643 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:14.807709 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:14.807829 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:14.807831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:14.808148 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:14.808212 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.808214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.808822 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.808823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.809400 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.809403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.809873 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.809874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.809931 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.810382 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:14.810391 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:14.811873 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:14.813159 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.813165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.813167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.813341 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.813514 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.813515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.813515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.813600 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.813812 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:14.813816 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:14.813883 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 23:21:14.813967 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:21:14.814124 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260928 23:21:14.814867 23163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:21:14.820679 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.815169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.0.98.61:33731)
0928 23:21:14.815209 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.815210 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.815217 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:21:14.815240 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.815241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.815250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.815301 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.815496 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.816042 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.816684 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.816937 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.816984 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.818553 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.818554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.818555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.818560 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.819786 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.820094 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.820095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.820095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.820148 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.820284 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.820285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.820285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.820405 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.820473 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.820558 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.820620 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:21:14.820689 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.815111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33731 (local address 127.0.0.1:46354)
0928 23:21:14.815145 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.815153 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:14.815180 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:14.815302 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:14.815302 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:14.815406 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:14.815434 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.815435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.816120 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.816121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.816620 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.816621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.817005 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.817006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.817056 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.817385 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:14.817390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:14.818463 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:14.819856 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.819856 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.819857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.820031 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.820209 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.820209 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.820209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.820249 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.820461 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:14.820462 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:14.820491 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 23:21:14.820532 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 23:21:14.820605 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:21:14.821015 23163 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:21:14.821604 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.815043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.0.98.62:40679)
0928 23:21:14.815081 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.815083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.815091 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:21:14.815129 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.815130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.815139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.815188 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.815706 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.816169 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.816821 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.817024 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.817082 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.819117 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.819118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.819118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.819125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.820514 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.820864 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.820865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.820865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.820918 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.821129 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.821130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.821131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.821254 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.821429 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.821490 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.821549 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:21:14.827178 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.821234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52392)
0928 23:21:14.821274 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.821283 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:14.821316 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:14.821424 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:14.821425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:14.821552 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:14.821581 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.821583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.822357 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.822358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.822809 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.822810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.823138 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.823138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.823181 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.823509 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:14.823514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:14.824857 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:14.826320 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.826321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.826321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.826485 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.826651 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.826651 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.826652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.826701 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.826892 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:14.826893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:14.826935 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:21:14.826994 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:21:14.827083 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 23:21:14.827243 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.821285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.0.98.60:38235)
0928 23:21:14.821314 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.821315 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.821322 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:21:14.821365 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.821366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.821375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.821421 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.821649 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.822287 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.822876 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.823082 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.823135 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.824958 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.824959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.824959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.824966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.826240 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.826552 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.826553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.826553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.826599 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.826749 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.826750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.826750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.826841 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.826893 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.827018 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.827132 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":16}
I20260928 23:21:14.828594 23163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa: Bootstrap starting.
I20260928 23:21:14.828897 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.822271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.0.98.61:33731)
0928 23:21:14.822305 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.822307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.822347 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:21:14.822417 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.822418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.822427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.822469 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.822853 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.823297 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.824152 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.824522 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.824594 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.826680 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.826681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.826681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.826687 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.827999 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.828271 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.828272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.828272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.828320 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.828470 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.828471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.828471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.828569 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.828719 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.828753 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.828816 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:21:14.829516 23163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:14.829777 23163 log.cc:824] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:14.830132 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:21:14.830528 23163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa: No bootstrap required, opened a new log
I20260928 23:21:14.831054 22987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0713b764e0844703beb3d045b1b3bd45"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41492
I20260928 23:21:14.832042 23163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } }
I20260928 23:21:14.832161 23163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880687de171345f2a46d82abacca21fa, State: Initialized, Role: FOLLOWER
I20260928 23:21:14.832249 23163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } }
I20260928 23:21:14.832496 23169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } } }
I20260928 23:21:14.832544 23163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:14.832576 23169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:21:14.834295 23180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:14.834367 23180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:14.834615 23095 master_runner.cc:186] Error getting master registration for 127.0.98.62:40679: OK, instance_id { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" instance_seqno: 1790637674511294 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:21:14.837255 23095 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:21:14.851909 23013 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } attempt: 2
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:21:14.858424 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.852214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52398)
0928 23:21:14.852254 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.852282 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:14.852326 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:14.852453 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:14.852454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:14.852614 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:14.852654 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.852656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.853248 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.853250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.853739 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.853740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.854052 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.854053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.854106 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.854512 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:14.854517 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:14.855679 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:14.857405 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.857406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.857407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.857612 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.857788 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.857789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.857789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.857842 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.858126 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:14.858127 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:14.858169 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:21:14.858228 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:21:14.858316 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 23:21:14.858428 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.852287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.0.98.60:38235)
0928 23:21:14.852349 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.852351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.852370 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:21:14.852393 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.852394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.852403 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.852444 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.852726 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.853182 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.853813 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.853994 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.854049 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.855772 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.855774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.855775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.855782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.857315 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.857673 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.857674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.857674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.857732 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.857891 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.857892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.857892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.858067 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.858135 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.858260 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.858335 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:21:14.860056 23013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45: Bootstrap starting.
I20260928 23:21:14.860800 23013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:14.861099 23013 log.cc:824] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:14.861935 23013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45: No bootstrap required, opened a new log
I20260928 23:21:14.863276 23013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } }
I20260928 23:21:14.863420 23013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0713b764e0844703beb3d045b1b3bd45, State: Initialized, Role: FOLLOWER
I20260928 23:21:14.863504 23013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } }
I20260928 23:21:14.863776 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } } }
I20260928 23:21:14.863832 23013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:14.863867 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:14.863966 22987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 23:21:14.864027 22987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:14.864709 22987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } }
I20260928 23:21:14.865401 22987 leader_election.cc:288] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [CANDIDATE]: Term 1 election: Requested vote from peers 3a9afaa74caa42abaf47468e1fe624a2 (127.0.98.61:33731), 880687de171345f2a46d82abacca21fa (127.0.98.60:38235)
W20260928 23:21:14.865857 23194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:14.865906 23194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:14.865835 23137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0713b764e0844703beb3d045b1b3bd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "880687de171345f2a46d82abacca21fa"
I20260928 23:21:14.865978 23137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:14.866820 23137 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0713b764e0844703beb3d045b1b3bd45 in term 1.
I20260928 23:21:14.867053 22961 leader_election.cc:302] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0713b764e0844703beb3d045b1b3bd45, 880687de171345f2a46d82abacca21fa; no voters:
I20260928 23:21:14.867157 23183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:21:14.867278 23183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [term 1 LEADER]: Becoming Leader. State: Replica: 0713b764e0844703beb3d045b1b3bd45, State: Running, Role: LEADER
I20260928 23:21:14.867347 23183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } }
I20260928 23:21:14.867759 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0713b764e0844703beb3d045b1b3bd45. Latest consensus state: current_term: 1 leader_uuid: "0713b764e0844703beb3d045b1b3bd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } } }
I20260928 23:21:14.867820 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:14.868346 22945 master_runner.cc:186] Error getting master registration for 127.0.98.61:33731: OK, instance_id { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" instance_seqno: 1790637674652761 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:21:14.870510 22945 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:21:14.871729 23201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:21:14.872087 23201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:21:14.875109 23137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [term 1 FOLLOWER]: Refusing update from remote peer 0713b764e0844703beb3d045b1b3bd45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:21:14.875524 23183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:21:14.876855 23169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0713b764e0844703beb3d045b1b3bd45. Latest consensus state: current_term: 1 leader_uuid: "0713b764e0844703beb3d045b1b3bd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } } }
I20260928 23:21:14.876945 23169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:14.884771 23088 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } attempt: 3
I20260928 23:21:14.885893 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0713b764e0844703beb3d045b1b3bd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } } }
I20260928 23:21:14.886011 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:14.886482 23169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0713b764e0844703beb3d045b1b3bd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } } }
I20260928 23:21:14.886569 23169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:14.886792 23203 mvcc.cc:205] Tried to move back new op lower bound from 7334451916285460480 to 7334451916257406976. Current Snapshot: MvccSnapshot[applied={T|T < 7334451916285460480}]
I20260928 23:21:14.886974 23207 mvcc.cc:205] Tried to move back new op lower bound from 7334451916285460480 to 7334451916257406976. Current Snapshot: MvccSnapshot[applied={T|T < 7334451916285460480}]
I20260928 23:21:14.888336 23201 catalog_manager.cc:1347] Generated new cluster ID: c4b4702eb36042dcb1d3ff78884a0b36
I20260928 23:21:14.888382 23201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:21:14.900735 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.873917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.0.98.60:38235)
0928 23:21:14.873954 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.873956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.875032 (+ 1076us) server_negotiation.cc:407] Connection header received
0928 23:21:14.875209 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.875210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.875234 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.875264 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.877477 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.878245 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.884638 (+ 6393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.884961 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.885002 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.892407 (+ 7405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.892420 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.892421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.892429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.893975 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.898080 (+ 4105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.898081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.898082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.898151 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.899121 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.899122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.899122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.899247 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.899324 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.900561 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.900642 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:21:14.901499 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.874280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.0.98.61:33731)
0928 23:21:14.874320 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.874321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.876931 (+ 2610us) server_negotiation.cc:407] Connection header received
0928 23:21:14.877230 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.877231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.877253 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.877282 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.880888 (+ 3606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.881598 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.887231 (+ 5633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.887989 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.888034 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.894758 (+ 6724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.894759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.894760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.894767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.896989 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.898604 (+ 1615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.898605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.898605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.898666 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.899519 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.899520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.899520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.899681 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.899755 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.901331 (+ 1576us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.901408 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:21:14.902172 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.874073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.0.98.62:40679)
0928 23:21:14.874108 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.874109 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.876478 (+ 2369us) server_negotiation.cc:407] Connection header received
0928 23:21:14.876742 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.876743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.876765 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.876792 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.881759 (+ 4967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.882476 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.888137 (+ 5661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.888426 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.888464 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.894358 (+ 5894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.894360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.894361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.894379 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.896248 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.898452 (+ 2204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.898453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.898453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.898516 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.899905 (+ 1389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.899906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.899906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.900033 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.900104 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.902005 (+ 1901us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.902083 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 23:21:14 dist-test-slave-djkr krb5kdc[22932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:21:14.907977 23201 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:21:14.909252 23201 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:21:14.909734 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.885014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38235 (local address 127.0.0.1:52422)
0928 23:21:14.885095 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 23:21:14.885106 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:21:14.885140 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:21:14.897472 (+ 12332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:21:14.897473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:21:14.897636 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:21:14.897681 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.897683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.900396 (+ 2713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.900397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.901211 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:21:14.901214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.902586 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.902588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:21:14.902660 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.903039 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:21:14.903044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:21:14.904479 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:21:14.908680 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.908681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.908681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.908875 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.909112 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:21:14.909112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:21:14.909113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:21:14.909176 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:21:14.909449 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:21:14.909450 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:21:14.909494 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:21:14.909550 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 23:21:14.909644 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 23:21:14.910111 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:14.885168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.0.98.60:38235)
0928 23:21:14.897167 (+ 11999us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.897169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.897177 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:21:14.897197 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.897198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.897296 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.897828 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.897969 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.900217 (+ 2248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.901384 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:14.902412 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:14.902809 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:14.906340 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:21:14.906341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:21:14.906341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:21:14.906348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:21:14.908583 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.908973 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.908974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.908974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.909034 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:21:14.909242 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:21:14.909243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:21:14.909243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:21:14.909369 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:21:14.909890 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:14.909933 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 23:21:14.910022 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11891,"thread_start_us":80,"threads_started":1}
I20260928 23:21:14.911598 23088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2: Bootstrap starting.
I20260928 23:21:14.912336 23088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2: Neither blocks nor log segments found. Creating new log.
I20260928 23:21:14.912649 23088 log.cc:824] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:21:14.913697 23088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2: No bootstrap required, opened a new log
I20260928 23:21:14.915797 23088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } }
I20260928 23:21:14.915935 23088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9afaa74caa42abaf47468e1fe624a2, State: Initialized, Role: FOLLOWER
I20260928 23:21:14.916028 23088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } }
I20260928 23:21:14.916348 23088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:21:14.916465 23061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:21:14.917326 23061 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [term 1 FOLLOWER]: Refusing update from remote peer 0713b764e0844703beb3d045b1b3bd45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:21:14.917505 23062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0713b764e0844703beb3d045b1b3bd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3a9afaa74caa42abaf47468e1fe624a2"
I20260928 23:21:14.917621 23211 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:21:14.918216 23062 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0713b764e0844703beb3d045b1b3bd45 in term 1.
I20260928 23:21:14.920662 23214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0713b764e0844703beb3d045b1b3bd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } } }
I20260928 23:21:14.920745 23214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:14.920797 23214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0713b764e0844703beb3d045b1b3bd45. Latest consensus state: current_term: 1 leader_uuid: "0713b764e0844703beb3d045b1b3bd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } } }
I20260928 23:21:14.920840 23214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:21:14.920871 23214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0713b764e0844703beb3d045b1b3bd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } } }
I20260928 23:21:14.920914 23214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:21:14.924101 23225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:21:14.924163 23225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:21:14.925133 23227 mvcc.cc:205] Tried to move back new op lower bound from 7334451916417441792 to 7334451916257406976. Current Snapshot: MvccSnapshot[applied={T|T < 7334451916285460480}]
I20260928 23:21:14.932175 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0713b764e0844703beb3d045b1b3bd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0713b764e0844703beb3d045b1b3bd45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } peers { permanent_uuid: "3a9afaa74caa42abaf47468e1fe624a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33731 } } peers { permanent_uuid: "880687de171345f2a46d82abacca21fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38235 } } }
I20260928 23:21:14.932268 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45 [sys.catalog]: This master's current role is: LEADER
I20260928 23:21:14.939503 23201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0713b764e0844703beb3d045b1b3bd45: Generated new TSK 0
I20260928 23:21:14.939822 23201 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:21:14.966172 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:14.964513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.0.98.61:33731)
0928 23:21:14.964554 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.964556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.965477 (+ 921us) server_negotiation.cc:407] Connection header received
0928 23:21:14.965588 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.965589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.965600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.965628 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.966101 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46362: BlockingRecv error: recv got EOF from 127.0.0.1:46362 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 23:21:14.966908 23016 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:14.964875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.0.98.62:40679)
0928 23:21:14.964915 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.964917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.966422 (+ 1505us) server_negotiation.cc:407] Connection header received
0928 23:21:14.966521 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.966522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.966533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.966563 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.966845 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41510: BlockingRecv error: recv got EOF from 127.0.0.1:41510 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 23:21:14.967602 23209 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:14.965215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.0.98.60:38235)
0928 23:21:14.965253 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.965255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.967127 (+ 1872us) server_negotiation.cc:407] Connection header received
0928 23:21:14.967217 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.967218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.967228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.967254 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.967528 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52426: BlockingRecv error: recv got EOF from 127.0.0.1:52426 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 23:21:14.969775 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:14.968716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.0.98.61:33731)
0928 23:21:14.968748 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.968750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.969302 (+ 552us) server_negotiation.cc:407] Connection header received
0928 23:21:14.969402 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.969403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.969422 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.969449 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.969735 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46368: BlockingRecv error: recv got EOF from 127.0.0.1:46368 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 23:21:14.972615 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:14.971889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.0.98.61:33731)
0928 23:21:14.971920 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.971922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.972168 (+ 246us) server_negotiation.cc:407] Connection header received
0928 23:21:14.972256 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.972257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.972266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.972292 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.972574 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46382: BlockingRecv error: recv got EOF from 127.0.0.1:46382 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 23:21:14.976310 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:14.975595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.0.98.61:33731)
0928 23:21:14.975626 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.975627 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.975850 (+ 223us) server_negotiation.cc:407] Connection header received
0928 23:21:14.975932 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.975933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.975942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.975994 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.976271 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46396: BlockingRecv error: recv got EOF from 127.0.0.1:46396 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 23:21:14.982102 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:14.981292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.0.98.61:33731)
0928 23:21:14.981324 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.981325 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.981557 (+ 232us) server_negotiation.cc:407] Connection header received
0928 23:21:14.981645 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.981646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.981655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.981681 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.982063 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46400: BlockingRecv error: recv got EOF from 127.0.0.1:46400 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 23:21:14.991845 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:14.991086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.0.98.61:33731)
0928 23:21:14.991116 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:14.991118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:14.991335 (+ 217us) server_negotiation.cc:407] Connection header received
0928 23:21:14.991420 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:14.991421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:14.991430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:14.991456 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:14.991799 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46408: BlockingRecv error: recv got EOF from 127.0.0.1:46408 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260928 23:21:15.009769 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:15.008893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.0.98.61:33731)
0928 23:21:15.008934 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:15.008936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:15.009060 (+ 124us) server_negotiation.cc:407] Connection header received
0928 23:21:15.009107 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:15.009108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:15.009119 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:15.009165 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:15.009715 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46424: BlockingRecv error: recv got EOF from 127.0.0.1:46424 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 23:21:15.043092 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:15.042415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.0.98.61:33731)
0928 23:21:15.042458 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:15.042459 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:15.042598 (+ 139us) server_negotiation.cc:407] Connection header received
0928 23:21:15.042660 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:15.042661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:15.042672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:15.042712 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:15.043032 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46434: BlockingRecv error: recv got EOF from 127.0.0.1:46434 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 23:21:15.108862 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:15.108229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.0.98.61:33731)
0928 23:21:15.108286 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:15.108288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:15.108367 (+ 79us) server_negotiation.cc:407] Connection header received
0928 23:21:15.108447 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:15.108448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:15.108458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:15.108506 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:15.108801 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46450: BlockingRecv error: recv got EOF from 127.0.0.1:46450 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 23:21:15.238622 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:15.237978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.0.98.61:33731)
0928 23:21:15.238040 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:15.238042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:15.238100 (+ 58us) server_negotiation.cc:407] Connection header received
0928 23:21:15.238171 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:15.238172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:15.238184 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:15.238234 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:15.238559 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46452: BlockingRecv error: recv got EOF from 127.0.0.1:46452 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 23:21:15.496265 23092 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:21:15.495717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.0.98.61:33731)
0928 23:21:15.495780 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:15.495782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:15.495796 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:21:15.495857 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:15.495858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:15.495868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:15.495917 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:21:15.496206 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46462: BlockingRecv error: recv got EOF from 127.0.0.1:46462 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:21:15.834834 23180 catalog_manager.cc:1259] Loaded cluster ID: c4b4702eb36042dcb1d3ff78884a0b36
I20260928 23:21:15.834904 23180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa: loading cluster ID for follower catalog manager: success
I20260928 23:21:15.835757 23180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa: acquiring CA information for follower catalog manager: success
I20260928 23:21:15.836058 23180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 880687de171345f2a46d82abacca21fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:21:15.924615 23225 catalog_manager.cc:1259] Loaded cluster ID: c4b4702eb36042dcb1d3ff78884a0b36
I20260928 23:21:15.924692 23225 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2: loading cluster ID for follower catalog manager: success
I20260928 23:21:15.925519 23225 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2: acquiring CA information for follower catalog manager: success
I20260928 23:21:15.925805 23225 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a9afaa74caa42abaf47468e1fe624a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:21:16.012060 23297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:16.009500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.98.61:33731)
0928 23:21:16.009609 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:16.009611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:16.009666 (+ 55us) server_negotiation.cc:407] Connection header received
0928 23:21:16.009710 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:16.009711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:16.009761 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:16.009834 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:21:16.010005 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:16.010510 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:16.011296 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:16.011503 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:16.011572 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:16.011750 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:21:16.011862 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:21:16.011923 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:16.011956 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 23:21:16.011975 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260928 23:21:16.016091 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:16.013763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.0.98.60:38235)
0928 23:21:16.013911 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:16.013913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:16.013922 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:21:16.013957 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:16.013958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:16.013994 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:16.014081 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:21:16.014167 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:16.014648 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:16.015330 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:16.015522 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:16.015623 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:16.015765 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:21:16.015869 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:21:16.015921 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:16.015979 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 23:21:16.016000 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260928 23:21:16.020510 23297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:16.017597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.0.98.61:33731)
0928 23:21:16.017634 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:16.017635 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:16.017856 (+ 221us) server_negotiation.cc:407] Connection header received
0928 23:21:16.017987 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:16.017988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:16.017999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:16.018050 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:21:16.018236 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:16.018946 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:16.019810 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:16.020099 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:16.020158 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:16.020265 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:21:16.020297 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:21:16.020338 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:16.020417 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 23:21:16.020441 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:21:16.020547 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:16.017756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.0.98.62:40679)
0928 23:21:16.017849 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:16.017851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:16.017858 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:21:16.017945 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:16.017947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:16.018000 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:16.018087 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:21:16.018187 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:16.018673 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:16.019624 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:16.019908 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:16.019969 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:16.020114 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:21:16.020229 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:21:16.020283 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:16.020432 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 23:21:16.020460 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":49,"threads_started":1}
I20260928 23:21:16.020623 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:21:16.017693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.0.98.60:38235)
0928 23:21:16.017737 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:21:16.017739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:21:16.017962 (+ 223us) server_negotiation.cc:407] Connection header received
0928 23:21:16.017984 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:21:16.017985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:21:16.017994 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:21:16.018044 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:21:16.018231 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:16.018946 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:16.019890 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:21:16.020140 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:21:16.020196 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:21:16.020384 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:21:16.020422 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:21:16.020465 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:21:16.020545 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 23:21:16.020566 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:21:16.022236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 22945
I20260928 23:21:16.026718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 23020
I20260928 23:21:16.031039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNOOpNT/build/release/bin/kudu with pid 23095
2026-09-28T23:21:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2701 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5385 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (258401 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST